A novel continuous looseness model of a Jeffcott rotor system with pedestal looseness fault was proposed, which differs from the conventional piecewise-linear models. Based on the present continuous model, the nonlinear differential equation of rotor system was established, and IHB method was adopted to solve this nonlinear differential equation. The periodic solution was obtained by means of a simulation example based on the iterative algorithm. Based on these results, the time domain waveforms, amplitude spectrograms, phase diagrams and the Poincare sectional drawings could be obtained to analyze the dynamic characteristics of this failing system. The present study has proved that the new looseness model is reliable and IHB method is feasible to solve such a nonlinear system. All the results of this paper could provide solid theoretical basis for the diagnosis of looseness fault.