Since the 1980s, grassland fragmentation has become increasingly serious in pastoral areas of north China, and has become a crucial feature of grasslands after the grassland contract management system reform. Therefore, this study focusses on the practical effect of grassland fragmentation on herders in pastoral areas, particularly the impact of grassland fragmentation on the livestock production and management of herding households, which not only could provide theoretical support for the better improvement of subsequent land policies, but also provide more basis so as to achieve sustainable development in pastoral areas. In view of this, this paper takes the semi-arid pastoral areas of Otog Banner and Wushen Banner as an example, and empirically measures the impact of livestock efficiency and its mechanism by using the micro-survey data from 153 herding households. Firstly, the Data Envelopment Analysis-BCC (DEA-BCC) model and entropy weight-TOPSIS method were used to measure the livestock efficiency and grassland fragmentation degree of herding households. Secondly, the livestock efficiency was decomposed into technical efficiency and scale efficiency, and then the Tobit model was used to estimate the relationship between grassland fragmentation degree and efficiency of livestock. Finally, the forage expenses was used as a mediator variable to evaluate a mechanism about the impact of grassland fragmentation on livestock efficiency, then we analyzed the heterogeneity of mechanism and tested the robustness of the conclusions. The results showed that: (1) The grassland fragmentation degree has different impacts on the livestock efficiency, indicating that the technical efficiency fail to significantly improve when the degree of grassland fragmentation increase, but the impact on the scale efficiency is not significant. (2) The increase in grassland fragmentation degree would stimulate the soar of forage expenses of herding households, and then reduce the efficiency of livestock. Similarly, this conduction mechanism only has a significant impact on technical efficiency. (3) Using normalized difference vegetation index (NDVI) data for heterogeneity analysis, this study found that the impact of grassland fragmentation on livestock efficiency showed obvious heterogeneity. Specifically, the impact mechanism varied with different vegetation coverage, and the negative impact of grassland fragmentation on livestock efficiency was more obvious in research areas where has higher vegetation coverage. In conclusion, this study suggests that grassland ecological protection should be paid attention to. At the same time, pasture reintegration should be achieved to mitigate the impact of pasture fragmentation by advocating collective action among herders, conducting joint household management and promoting herders' self-governance and co-operation.