The non-circular planetary gear motor has the advantages of good sealing performance and stable output torque, which can meet the working requirements of harsh underground environment. The geometric model and finite element model of the non-circular planetary gear motor were established, and the simulation parameters of the model were set and the grid independence was verified. The flow field of a non-circular planetary gear motor was simulated and the velocity variation rule of the flow field at the oil film position was investigated. The flow rate and volumetric efficiency of Q100, Q160 and Q250 non-circular planetary gear motors under inlet pressure of 5~20MPa were studied. The flow rate and volumetric efficiency of Q250 non-circular planetary gear motor with oil film gap of 0.02mm, 0.03mm and 0.04mm were studied. The results of prototype test show that the curve of volumetric efficiency and torque relationship obtained by experiment and theory is basically the same, and the maximum error is only 5.76%, which verifies the accuracy of the simulation model. This research can provide support for the structural optimization design of non-circular planetary gear motors.