When the high speed precision punch is running, the internal mechanism of massless inertial force effect on the frame, which leads the imbalance in the frame and causes the vibration of punch. So in this case we can reduce the accuracy of parts dealing in mechanical processing and shorten the life of the use. In this paper, the dynamic unbalance force and vibration mode are analyzed, which obtained the best quality and the balance effect of the mass block, and the scheme of reducing vibration is put forward from the aspect of structure improvement. Firstly, in order to understand the force of the working process of the punch, the inertia force model is derived. Then using the parameters which including counterweight block and ground rod centroid location to do the analysis and comparison of inertial force. Finally, we used the derived inertial force as the boundary condition to do the analysis of the structure vibration characteristics in the finite element software ANSYS, and found out the improved scheme of partial structure. This study provides a theoretical basis for the design of punch press.