Due to their unmatched entropy, complexity, and security level, optical Physical Unclonable Functions (PUFs) currently receive a lot of interest in the literature. Despite the large body of existing works, however, one of their core features has never been quantified in detail, namely their physical unclonability. This paper tackles this fundamental and yet largely unaddressed issue. In simulations and/or experiments, the sensitivity of diffraction-based optical responses is investigated with respect to various small alterations such as variation in the position, size, and number of the scatterers, as well as perturbations in the spatial alignment between the physical unclonable function (PUF) and the measurement apparatus. Our analysis focuses on 2D optical PUFs because of their relevance in integrated applications and the need to reply to security concerns that can be raised when the physical structure of the geometry is accessible. Among the results of this study, the sensitivity analysis shows that a positional perturbation of scatterers on the order of \SI{30}{\nano\meter}, i.e., far below the wavelength of the probing laser light of \SI{632}{\nano\meter} wavelength, is sufficient to invalidate the PUF response and thus detect a forgery attempt. These results support and quantify the high adversarial efforts required to clone optical PUFs, even for 2D layouts.