In this paper, we propose a fast second-order approximation to the variable-order (VO) Caputo fractional derivative, which is developed based on $L2$-$1_\sigma$ formula and the exponential-sum-approximation technique. The fast evaluation method can achieve the second-order accuracy and further reduce the computational cost and the acting memory for the VO Caputo fractional derivative. This fast algorithm is applied to construct a relevant fast temporal second-order and spatial fourth-order scheme ($FL2$-$1_{\sigma}$ scheme) for the multi-dimensional VO time-fractional sub-diffusion equations. Theoretically, $FL2$-$1_{\sigma}$ scheme is proved to fulfill the similar properties of the coefficients as those of the well-studied $L2$-$1_\sigma$ scheme. Therefore, $FL2$-$1_{\sigma}$ scheme is strictly proved to be unconditionally stable and convergent. A sharp decrease in the computational cost and the acting memory is shown in the numerical examples to demonstrate the efficiency of the proposed method.