This paper is devoted to the error analysis of a time-spectral algorithm for fractional diffusion problems of order $\alpha$ ($0 < \alpha < 1$). The solution regularity in the Sobolev space is revisited, and new regularity results in the Besov space are established. A time-spectral algorithm is developed which adopts a standard spectral method and a conforming linear finite element method for temporal and spatial discretizations, respectively. Optimal error estimates are derived with nonsmooth data. Particularly, a sharp temporal convergence rate $1+2\alpha$ is shown theoretically and numerically.