Matrix Vector Multiplications are a dominant contributor to high compute, memory and power budgets of Deep Neural Networks. This work proposes two algorithm/hardware co-design techniques with low complexity Alphabet Set Multipliers as the energy-efficient base unit to approximate Multiply-Accumulate operations. The approaches involve designing novel fully digital Near-Memory and Compute-in-Memory hardware architectures and their corresponding Quantized-Hardware Aware Training methodologies. For CIFAR10 and ImageNet datasets, our results show energy benefits of about 50% on 4-bit 65nm all-digital design, compared to its standard von-Neumann counterpart, with only less than 1 to 2% accuracy degradation against full-precision ResNet and MobileNet baseline models.