In view of poor quality and low processing efficiency of the installation holes on the Carbon Fiber Reinforced Polymer(CFRP) composite parts of highspeed rail, a new machining process of longitudinaltorsional compound ultrasonic helical milling hole is put forward. Firstly, based on the principal of motion synthesis, the space motion track of tool nose and end edge was obtained, and the elliptical vibration machining characteristics of longitudinaltorsional compound ultrasonic helical milling were analyzed. Then, based on the transfer matrix method and the modal transformation principal of helical groove, the frequency equation of catenary transition compound horn with milling cutter was derived, and the longitudinaltorsional compound vibration of ultrasonic machining system was realized. Meanwhile, the finite element analysis(FEA) of the ultrasonic machining system was carried out by using Ansys software, and the influence law of the structural parameters on the frequency and amplitude ratio of the machining system was obtained. At last, the machining performance of the developed ultrasonic machining system is tested, and better machining results are obtained.