In the field of virtual assembly, improving user experience and interaction accuracy has been the focus of research. Based on using Azure Kinect human tracking technology to obtain hand information, SVM algorithm was studied to improve the accuracy of gesture recognition and the parameter model was further optimized through parameter optimizing. A virtual assembly system was developed on Unity3D platform, and the optimized gesture recognition was applied to the virtual assembly system. The main assembly functions were implemented, while speech recognition was added to assist the assembly. The experimental results show that the gesture recognition is more accurate and stable after the optimization by SVM algorithm, which improves the virtual assembly system’ s user experience and interaction accuracy.