Based on the development of shrub encroachment in arid and semi-arid grasslands, exploring the relationship between shrubs and herbs has important significance to deeply understand the mechanism of effects of shrub encroachment on grassland plant community characteristics. However, relatively few studies consider the effect grassland types and shrub patch size on the relationship between shrubs and herbs. We selected 18 sites with shrub-encroached of Caragana microphylla for field sampling in free grazing grasslands in typical steppe and desert steppe of Inner Mongolia, and investigated the plant community characteristics of different sizes of shrubs and adjacent interspaces. One-way analysis of variance (one-way ANOVA) and Least Significant Difference (LSD) were used to test the differences in plant community characteristics (height, density, aboveground biomass, litter mass) and plant diversity indexes (Shannon-Wiener index, Pielou index, richness index, community Sprensen similarity coefficient) inside and outside shrub patches with different sizes. The results were as follows. (1) Compared with interspace, the shrubs both in two grassland types produced and captured more litter and the height of herbaceous plants in the shrub patch was significantly increased by facilitation of shrub, but shrub occupied the living space and resources, which reduced the density and aboveground biomass of herbaceous plants in the patch. (2) Shrubs had obvious inhibitory effects on the aboveground biomass of perennial herbs, but had no significant effect on annual and biennial herbs in two grassland types (except for small shrubs in typical steppe). (3) There are differences in the effects of shrubs on herbaceous plant diversity in two grassland types. The diversity index and richness index in the shrubs in typical steppe were lower than those outside the shrubs, the evenness index did not change significantly, and the expansion of shrub patch had a negative impact on the richness of herb species. In desert steppe, with relatively scarce resources, shrubs did not change the diversity index of herbs, but decreased the richness index. We also found that the evenness index of herbs decreased in the large shrub patches. (4) The similarity of community between shrub patch and interspace showed an exponential downward trend with the expansion of shrubs, the similarity of community between small shrub patch and interspace was the highest in two grassland types. This study showed that shrubs had both facilitation and competition effects on herbs, and these effects were affected by grassland type and patch size. This was important to predict and regulate the dynamics of plant community in shrub-encroached grasslands, and provided basis for the optimal management of grassland.