Cylindrical gear with variable hyperbolic circular-arc-tooth-trace (VH-CATT) is a new type of gear that applies the arc tooth line to the gear tooth line. Due to the lack of special processing machine tool, the adjustment of the cutting parameters is more complicated. In order to solve this problem, ABAQUS was used to simulate the cutting process according to its meshing principle and processing process; the obtained data were used to establish a cutting force prediction model based on orthogonal experiment method. By using the whale algorithm, a function optimization model with processing efficiency and smaller cutting force as the objective was established, the model was transformed into a single objective function optimization model through the weighted summation method and normalization processing, and the optimized cutting parameters were obtained by using the whale algorithm. The results show that the cutting parameters can be optimized very well by using the proposed single objective function optimization model to obtain better machining effect; the optimized cutting parameters are spindle speed n=189.3 r/min, feed per tooth fz=0.046 mm, depth of cut ap=1.89 mm.