Powder bed fusion additive manufacturing (PBFAM) of metals has the potential to enable new paradigms of product design, manufacturing and supply chains while accelerating the realization of new technologies in the medical, aerospace, and other industries. Currently, wider adoption of PBFAM is held back by difficulty in part qualification, high production costs and low production rates, as extensive process tuning, post-processing, and inspection are required before a final part can be produced and deployed. Physics-based modeling and predictive simulation of PBFAM offers the potential to advance fundamental understanding of physical mechanisms that initiate process instabilities and cause defects. In turn, these insights can help link process and feedstock parameters with resulting part and material properties, thereby predicting optimal processing conditions and inspiring the development of improved processing hardware, strategies and materials. This work presents recent developments of our research team in the modeling of metal PBFAM processes spanning length scales, namely mesoscale powder modeling, mesoscale melt pool modeling, macroscale thermo-solid-mechanical modeling and microstructure modeling. Ongoing work in experimental validation of these models is also summarized. In conclusion, we discuss the interplay of these individual submodels within an integrated overall modeling approach, along with future research directions.