We devise a fast algorithm for the gradient of the long-time-average statistics of chaotic systems, with cost almost independent of the number of parameters. It runs on one sample orbit; its cost is linear to the unstable dimension. The algorithm is based on two theoretical results in this paper: the adjoint shadowing lemma for the shadowing contribution and the fast adjoint formula for the unstable divergence in the linear response.