In the process of end milling, chatter is considered as one of the most important causes of instability in the machining processes. It would lead to very poor surface finish, low material removal rate, severe tool wear, and noisy workplace. Chatter vibration can be detected by most chatter detection system, however, it already has serous effects on the surface quality of workpieces as well as the cutting tools when it occures. Therefore, chatter detection system should find chatter chracteristics in the early state. In the chatteremerging processes, the amplitude of vibration signal increases in time domain, and the energy spectrum density of vibration signal transfers from high frequency to low frequency in frequency domain. A method to catch the chatter features was proposed in the transition state by considering the two vibration signal characteristics. The energy ratio and the singular spectrum entropy ratio of vibration signal in the chatteremerging frequency band were extracted as two chatter features. An artificial neural network (ANN) model was developed to identify the cutting of chatter characertistics. The chatter detection system, consisting of the feature extraction and identification, can accurately distinguish the stable, transition and chatter states in end milling processes.