We revisit the celebrated Kohn-Vogelius penalty method and discuss how to use it for the unique continuation problem where data is given in the bulk of the domain. We then show that the primal-dual mixed finite element methods for the elliptic Cauchy problem introduced in \cite{BLO18} (\emph{E. Burman, M. Larson, L. Oksanen, Primal-dual mixed finite element methods for the elliptic Cauchy problem, SIAM J. Num. Anal., 56 (6), 2018}) can be interpreted as a Kohn-Vogelius penalty method and modify it to allow for unique continuation using data in the bulk. We prove that the resulting linear system is invertible for all data. Then we show that by introducing a singularly perturbed Robin condition on the discrete level sufficient regularization is obtained so that error estimates can be shown using conditional stability. Finally we show how the method can be used for the identification of the diffusivity coefficient in a second order elliptic operator with partial data. Some numerical examples are presented showing the performance of the method for unique continuation and for impedance computed tomography with partial data.