A dynamic model of five degrees of freedom(5-DOF) angular contact ball bearingrotor system is established in order to study the nonlinear behavior of the rotor system. The dynamic differential equations were derived with the relation between the reaction of the angular contact ball bearing and displacement of the rotor journal centre. The RungeKutta method was applied to solve the equations. The nonlinear behavior of the rotor system applied on axial force was mainly discussed. The result shows that: the angular contact ball bearingrotor system is obviously nonlinear; the response of the system develops from periodic motion, period doubling motion, quasi periodic motion to chaotic motion with the increase of the axial force. The bearing stiffness is changed and the dynamic behavior of the two journal centres of the rotor are extremely different due to the action of axial force.