Grassland is an important and terrestrial ecosystem and one of the most widely distributed ecosystems in the world. In the context of climate change, grassland has a significant impact on global carbon source /sink dynamics and carbon cycling. The focus of the present study was the grassland vegetation of the Loess Plateau. We analyzed the effects of natural and degraded grassland grazing pre- and post-prohibition to combine the policy of returning farmland to forest or grassland and grazing prohibition. Using multi-point transect surveys and long-term fixed monitoring sites, plant distribution, leaf litter, and distribution and zonation of below ground carbon density of different types of grassland in the Loess Plateau were recorded. The temporal-spatial variability of biomass and carbon density with respect to the altitudinal and precipitation gradient was studied in several degraded grasslands and an enclosed grassland. The main driving factors of carbon density change and mechanisms underlying the distribution pattern of carbon density were also analyzed. The results indicated that the biomass and carbon density showed a tendency to exponentially increase with precipitation amount, and also significantly decrease with altitude from northwest to southeast in the degraded grassland, early enclosed grassland and enclosed grassland for 11 years among four types of grasslands. There was a significant linear relationship between biomass and carbon density. In each type of grassland, the amount of biomass of above ground and below ground carbon density was present in the following order: desert steppe< hills typical steppe< plateau steppe< meadow steppe. Carbon density of plants, leaf litter, and roots in grassland where grazing has been prohibited for 11 years was 7.066 t/hm² for desert steppe, 8.080 t/hm² for hilly steppe, 15.319 t/hm² for plateau steppe and 20.982 t/hm² for meadow steppe, which were 14.8 times, 8.33 times, 6.5 times, and 15.88 times higher than degraded grassland, respectively. Our results demonstrated that grazing prohibition can not only restore vegetation and increase biomass in the grassland, but also significantly improve grassland productivity and the potential for carbon sequestration. The arid climate and grassland degradation are critical factors that influence the biomass and carbon density of grassland. Study of biomass and carbon density change following grazing prohibition can make an important contribution towards the analysis of the effects of global climate change on grassland in the Loess Plateau.