Using RecurDyn and AMESim simulation software, a mechanicalhydraulic coupled dynamics simulation model of hydraulic shifting mechanism in a certain transmission system is established, and the reliability of the model is verified through experiment. In order to solve the problem of excessive calculation in the complex systems stochastic dynamics while the Monte Carlo method was used for reliability estimation, the method integrated with dynamic model & agent model & Monte Carlo method was proposed to estimate the operation reliability based on Isight platform. The artificial neural network agent model (RBF Model) with higher fitting accuracy was selected and the stochastic dynamics simulation solution and operating reliability estimation of the hydraulic shift system were completed. The results can provide references for evaluating the operational reliability of complex coupled system.