When the electromagnetic wave is incident on the periodic structures, in addition to the scattering field, some propagating modes that are traveling in the periodic medium could be generated. In the present paper, we study the calculation of propagating modes. We formulate the problem as a nonlinear eigenvalue problem in an unbounded periodic domain. Then we use perfectly matched layers to truncate the unbounded domain, recast the problem to a quadratic eigenvalue problem, and prove the approximation property of the truncation. Finally, we formulate the quadratic eigenvalue problem to a general eigenvalue problem, use the finite element method to discrete the truncation problem, and show numerical examples to verify theoretical results.