Understory herbs make up a basic functional layer of plantation ecosystems, and have great significance for their health, stability, and function. Aboveground biomass and species diversity of understory herbs in plantations of different ages and their relationships with stand structure and soil nutrient availability were compared and evaluated in representative sandy and loess hilly areas on the Loess Plateau, which is important for the rational assessment of future shelterbelts and ecological restoration sites. The results showed that: (1) the understory herb average aboveground biomass in sandy area (23.64 g/m²) was significantly lower than that of loess hilly areas (44.53 g/m²). As the duration of afforestation efforts increased, the understory herb aboveground biomass and species diversity in the sandy areas increased consistently, while in loess hilly areas, they declined gradually. (2) Soil available nitrogen and organic carbon in sandy areas increased gradually as duration of afforestation efforts increased, while in the loess hilly areas, they showed no obvious trend. (3) The increase in understory herb aboveground biomass and species diversity of younger plantations in sandy areas was a direct benefit of improvements in soil nutrient levels. In contrast, the understory herb aboveground biomass and species diversity decline of young plantations in the loess hilly areas was mainly caused by increases in canopy density. The plantation restoration efforts in sandy areas promoted the development of understory herbs. However, the understory herb layers of young plantations in loess hilly areas were severely degraded. These findings imply that shelterbelt construction is necessary for sandy areas, while natural restoration is more suitable for loess hilly areas.