An average healthy person does not perceive the world in just black and white. Moreover, the perceived world is not composed of pixels and through vision humans perceive structures. However, the acquisition and display systems discretize the world. Therefore, we need to consider pixels, structures and colors to model the quality of experience. Quality assessment methods use the pixel-wise and structural metrics whereas color science approaches use the patch-based color differences. In this work, we combine these approaches by extending CIEDE2000 formula with perceptual color difference to assess image quality. We examine how perceptual color difference-based metric (PCDM) performs compared to PSNR, CIEDE2000, SSIM, MS-SSIM and CW-SSIM on the LIVE database. In terms of linear correlation, PCDM obtains compatible results under white noise (97.9%), Jpeg (95.9%) and Jp2k (95.6%) with an overall correlation of 92.7%. We also show that PCDM captures color-based artifacts that can not be captured by structure-based metrics.