In view of the low efficiency of manual sorting fabric, high labor costs, and the problem of physical harm to the worker, a method of the machine vision combined with a robot to replace the manual sorting fabric is put forward, so as to solve the weakness of manual sorting fabric effectively. Firstly, the color classification of RBF neural network was constructed to classify different colored fabrics, and threshold segmentation was applied to segment the target fabric. Secondly, the crawl point of the robot was obtained by using the method of traverse the center diffusion. Finally, the camera calibrated and robot trajectory planning were used to achieve robot autonomous sorting for solid color fabrics. Experimental results show that the sorting accuracy of the proposed method can meet the application requirements of the textile industry.