Climate change and human activity are two main factors driving vegetation dynamics during grassland recovery or degradation. In this study, the net primary productivity (NPP) was selected as the index, the improved Carnegie-Ames-Stanford Approach (CASA) model and Thornthwaite Memorial model were used to estimate actual net primary productivity (ANPP) and potential net primary productivity (PNPP) determined by climatic conditions, respectively. The residual trend analysis (RESTREND) method was applied to analyze the residuals of potential and actual net primary productivity and further identify non-climatic effects on vegetation dynamics in four types of natural grassland, including meadow steppe, warm steppe, desert steppe and warm steppe desert of Ningxia. Results showed that the ANPP in 97.84% of the grassland increased from 2001 to 2019 generally. The PNPP of the whole region presented an increasing trend, revealing that climate change had a positive effect on vegetation recovery across the study area. During the grassland restoration, 61.68% of the recovered grassland resulted from climate change, while 38.32% of the recovered grassland from the combined effects of climate change and human activity. Anthropogenic activities were considered as the absolutely dominant influencing factor for vegetation degradation. The driving factors of vegetation dynamic showed differences in the four types of grassland. Climate change was the dominant factor for the recovery of warm meadow (68.94%) and warm steppe desert (70.51%). The combined effects of climate change and human activities were the main factors for the recovery of warm steppe (62.30%). The recovery of warm desert steppe resulted from the combined effects of climate change and human activities (97.93%). The improvement of hydrothermal conditions, especially the increases in precipitation, was the main influencing factor for grassland restoration in Ningxia. The implementation of ecological protection policies produced significantly positive effects on the restoration of grassland, and the unreasonable occupation of grassland was the main anthropogenic factor leading to grassland degradation.