Non-agricultural habitat is an important part of the agricultural landscape, and its complex structure and function play a vital role in maintaining and improving the biodiversity of the agricultural landscape. However, there is widespread evidence that the highly intensive global development of modern agriculture has led to the disappearance of many non-agricultural habitats, resulting in a simpler agricultural landscape. Therefore, the agro-ecosystem functions and services have been seriously damaged. Consequently, for the sustainable development of agriculture, studying the relationships between agricultural productivity and biodiversity, and landscape pattern and ecosystem functions and services are fundamental. To explore the structural attributes of non-agricultural landscape elements and the effect on plant species diversity in typical agricultural landscapes, we established a study area and developed 42 plots in Fengqiu County, Henan Province, which is a typical agricultural region in the lower reaches of the Yellow River. Within these plots, the vegetation in non-agricultural habitats (artificial forests, hedges, and ditches) was investigated using the Braun-Blanquet method. Based on remote sensing imagery and landscape pattern analysis, we extracted and classified the non-agricultural habitats of each sample within 1 km using ArcGIS 10.0, and analyzed the composition, structure, and spatial configuration of non-agricultural landscape elements. The results showed that 186 species belonging to 50 families and 164 genera were found in the study area, and six dominant families (Compositae, Gamineae, Leguminosae, Cruciferae, Labiatae, and Solanaceae) account for 51.6% of all species. The species diversity of artificial forests and hedges was relatively high, whereas the species composition of the ditches was different to the other habitats. In different types of non-agricultural landscape structure, the highest species richness of plants was found where the proportion of artificial forests was similar to hedge/ditches. The landscape metrics directly affect plant species diversity in different non-agricultural habitats, and the influence of landscape fragmentation and human disturbance is significant. From the results, to increase the quantity and quality of non-agricultural habitats in agricultural landscapes in the future, we should transform the non-agricultural landscape elements. New non-agricultural habitats should be developed, and the proportion of different types of habitats should be increased by adjusting the spatial structure and configuring different landscape elements. This would enhance the effective management and sustainable development of the agricultural landscape in the future.