Due to the high degree of ecological niche overlap, there is intense competition for light, nutrients, and water between early-stage herbaceous plants and tree seedlings during forest regeneration, significantly affecting the process of forest renewal. In this study, we focused on the seedlings of Quercus mongolica in Pangquangou Nature Reserve, Shanxi province. We conducted a pot experiment with four herbaceous competition modes (no competition, total competition, aboveground competition, and underground competition), two soil nutrient gradients (4 g/m² and 32 g/m²), and two soil nutrient distribution patterns (homogeneous and heterogeneous). The results indicated that: 1) Herbaceous competition and soil nutrient conditions had significant impact on seedling regeneration (P ＜ 0.05). Specifically, the differences in growth indicators of seedlings between the treatments of no competition and underground competition suggested the existence of underground competition effects, and the same applied to aboveground competition effects. 2) Based on a linear model, for seedling height, total leaf area, total leaf biomass, and surface area of three fine root diameter classes, the contribution rates of the main effect of herbaceous competition were 80.66%, 67.49%, 49.39%, 40.26%, 59.09%, and 53.26%, respectively. Compared to soil nutrient concentration, herbaceous competition had a higher forest regeneration effect. Under heterogeneous soil nutrient conditions, the interaction effect between herbaceous competition and soil nutrient concentration was even significantly greater than the main effect of soil concentration. The analysis of the components of herbaceous competition revealed that the contribution rate of underground competition effect was consistently dominant. 3) Based on structural equation modeling, the specific pathways of the forest regeneration effects of herbaceous competition and its underlying components could be revealed. In the treatments with no competition and full competition, the standardized regression coefficients of seedling regeneration potential on light absorption and nutrient absorption were 0.61 and 0.46, respectively. This indicated that herbaceous competition primarily affected the seedling regeneration process through its impact on light absorption. In the group analysis of the intrinsic components of herbaceous competition, the standardized coefficients corresponding to aboveground competition and belowground competition were 0.31 and 0.74, respectively. This indicates that the latter is the primary pathway of action. This study, based on quantitative evaluation of herbaceous competition and its intrinsic components' effects on forest regeneration, further explored the mechanisms involved. It deepens our understanding of their ecological effects in the early stages of forest regeneration and provides theoretical guidance for forest regeneration practices.