We propose in this work a fast numerical algorithm for solving the equation of radiative transfer (ERT) in isotropic media. The algorithm has two steps. In the first step, we derive an integral equation for the angularly averaged ERT solution by taking advantage of the isotropy of the scattering kernel, and solve the integral equation with a fast multipole method (FMM). In the second step, we solve a scattering-free transport equation to recover the original ERT solution. Numerical simulations are presented to demonstrate the performance of the algorithm for both homogeneous and inhomogeneous media.