We prove error estimates for a finite element approximation of viscoelastic dynamics based on continuous Galerkin in space and time, both in energy norm and in $L^2$ norm. The proof is based on an error representation formula using a discrete dual problem and a stability estimate involving the kinetic, elastic, and viscoelastic energies. To set up the dual error analysis and to prove the basic stability estimates, it is natural to formulate the problem as a system involving evolution equations for the viscoelastic stress, the displacements, and the velocities. The equations for the viscoelastic stress can, however, be solved analytically in terms of the deviatoric strain velocity, and therefore, the viscoelastic stress can be eliminated from the system, resulting in a system for displacements and velocities.