The spindle rotation error is an important factor affecting the machining accuracy. The generation and measurement methods of spindle rotation error were respectively analyzed and studied. On this basis, an optimization algorithm, namely lineshift fourpoint matrix algorithm, was used to separate the rotation error of the spindle to replace the traditional Fourier transform to realize the roundness error separation. Least square method and circular image error value were used to evaluate the separation of the spindle rotation error. By processing the measurement data, the obtained results reach higher accuracy, and the spindle rotation error can be effectively separated.