In order to explore the effect of altitude on the diversity of undergrowth plants in the cold temperate mountain area, we selected the eastern Greater Xing'an Mountains in Inner Mongolia by using the plot investigation method, and set six different altitude gradients: E1 (200-350 m), E2 (350-500 m), E3 (500-650 m), E4 (650-800 m), E5 (800-950 m), and E6 (950-1100 m). We investigated 165 forest sites to understand the composition of forest communities at different altitudes, and studied the relationship between the altitude difference in understory plant biodiversity, topographic factors, and dominant species in the arboreal layer and understory plant biodiversity. The results showed that: (1) there were 277 understory plant species belonging to 53 families and 135 genera, including 32 shrubs and 245 herbs. (2) The forest tree and shrub communities with different altitudes were in the order of Quercus Mongolia, Betula daturic, Corylus heterophylla, Lespedeza bicolor communities (E1-E2), Larix gmelina, Betula platyphylline, Corylus heterophylla, Lespedeza bicolor communities (E3), Larix gmelina, Betula platyphylline, Spiraea media, Spiraea silicified communities (E4), Larix gmelina, Betula platyphylla, Rhododendron dauricum, Vaccinium vitis-idaea communities (E5), Larix gmelinii, Betula platyphylla, Vaccinium vitis-idaea, Pinus pumila communities (E6). (3) The biodiversity of understory plant communities showed an obviously unimodal pattern with the increases of altitude. The herb layer was higher than the shrub layer. The shrub layer showed an increasing trend with the increasing altitude, while the herb layer showed a fluctuating decrease trend of first decreasing and then increasing and decreasing. (4) The replacement rate of species in herbaceous layer was much higher than that in shrub layer, with the replacement rate in shrub and herbaceous layer peaking in E3-E4 and E2-E3 gradients, respectively. The similarity index between shrub layer and herbage layer decreased first, then increased, and finally decreased. (5) The geographical factors and different dominant tree species could significantly affect the α diversity of understory plants. The α diversity of shrub layer was positively correlated with altitude and latitude, and negatively correlated with longitude and slope. The herbaceous layer was negatively correlated with altitude, but positively correlated with latitude and slope aspect. The effect of altitude on α diversity of herbaceous layer was higher than that of shrub layer. The canopy density was positively correlated with the α diversity of shrub layer and negatively correlated with the α diversity of herb layer. Larix gmelinii increased biodiversity in shrub layer, Betula dahurica decreased species richness in shrub layer, and Betula dahurica decreased species distribution evenness in herbaceous layer. Our results provided scientific basis for the protection and utilization of undergrowth plants in the Greater Xing'an Mountains and the theory of altitudinal distribution of understory plant diversity in the cold temperate mountains.