Depth learning method can automatically find better data to improve the performance of classifier. However, in computer vision tasks, such as gender recognition, it is difficult to learn directly from the entire image sometimes. Therefore, a new face recognition model based on local feature and depth neural network is proposed in this paper. Firstly, the model extracts several local features from the input image, and then these features are fed back to the depth neural network of the image, and then each local feature is classified according to the tag. Finally, a simple voting scheme is used to decide the whole image. The experiments of face gender classification are carried out on two face databases of FERET and CAS-PEAL-R1, and the results show that the proposed method is superior to other depth learning methods, and has better accuracy and stability.