This paper proposes a method to vibration analysis in order to on-line monitoring of milling process quality. Adapting envelope analysis to characterize the milling tool materials is an important contribution to the qualitative and quantitative characterization of milling capacity and a step by modeling the three-dimensional cutting process. An experimental protocol was designed and developed for the acquisition, processing and analyzing three-dimensional signal. The vibration envelope analysis is proposed to detect the cutting capacity of the tool with the optimization application of cutting parameters. The research is focused on Hilbert transform optimization to evaluate the dynamic behavior of the machine/ tool/workpiece.