In robot milling process, backoff error is an important factor influencing milling accuracy. Ballend cutter milling which is treated as quasistatic process was studied. A backoff error model for ballend cutter milling process was established based on the robot static elastic model and the ballend milling force model. Meanwhile, an optimization method for tool posture was presented using a genetic algorithm to reduce the backoff error in milling process. Finally, simulation was carried out to analyze the influence of cutting parameters, tool posture and robot joint stiffness on backoff error. The feasibility of the optimization method of tool posture is validated.