Because the traditional fabric detection algorithm has serious false detection and miss detection, and it is difficult to detect small defects, a fabric defect detection method based on sparse optimization was proposed. The fabric image was preprocessed to enhance the contrast of the image.Some nondefective fabric sample images were segmented, and the image patches were clustered into clusters by using K-means algorithm. A subdictionary was trained for each cluster and the image was reconstructed by using the optimized sparse representation model with the appropriate subdictionary. Finally, the residual image was generated, and the residual image was segmented by using the maximum entropy threshold method to detect the defect of the fabric. The experimental results show that the algorithm can be used to effectively detect various defects and minor defects of fabrics. Compared with other algorithms, the algorithm also has high detection accuracy.