Being the most important transmission part in various mechanical equipments, gear fault diagnosis is crucial in the longterm protection of equipment. The research status of fault feature extraction and pattern recognition method in recent years was described. The principle of cepstrum analysis was introduced and its advantages in the diagnosis of gearbox were expounded. The pitting corrosion in gearbox was diagnosed using the cepstrum analysis, combining amplitude spectrum and power spectrum analysis of the vibration signal of gearbox. Then the pitting gear was definitely confirmed. The experimental result has proved the efficiency and feasibility of cepstrum analysis.