According to the requirements of on-machine visual inspection of end mills, a telescopic inspection mechanism built on the side window of the machine tool was designed. The iterative method was used to quickly converge to obtain the initial threshold of the tool wear area, and the intra-class variance function was introduced to improve the maximum inter-class variance method,by which the defects of traditional algorithms, such as blurred boundaries, incomplete texture recognition and sensitivity to noise,were fixed.According to the rotation invariant characteristics of the gray moment, the Canny operator was fused to extract the sub-pixel wear edge, and based on the image plane, the wear contour model of the end mill was created, and the original contour of the cutting edge was reconstructed to realize the rapid detection of end mill wear.Finally, 5 end mills were selected to carry out milling experiments, and the measurement values of the detection system and the image measuring instrument were compared.The results show that the measurement deviation is less than 0.01 mm, and the average accuracy rate of each wear stage of the end mill in the life cycle reaches more than 93%, the comprehensive average accuracy rate reaches 95.98%, which proves the accuracy and stability of the detection system.