Ecologists have long been enthusiastic about alpha diversity in montane regions; however, the elevational patterns of beta diversity and underlying crucial processes are still rarely studied. Niche and neutral processes are known to be two key factors in shaping species composition in a community. Although it could be difficult to test independently because of covariability, these two factors could produce competing predictions concerning the pattern of beta diversity between communities with environmental divergence and geographic distance. Our goal was to discover the patterns of species turnover among sites in temperate montane forests, and determine to what extent environmental filtering and limitation of dispersal affect the processes of species assembly. Additionally, we tried to pinpoint the main factors associated with niche filtering, which dominantly control the local floristic composition of the understory. We investigated the coniferous forest communities along the elevational gradient of Luya Mountain, documenting all species encountered and environmental conditions in 1720 m × 30 m plots. For all pairwise sites, we calculated the Simpson's beta dissimilarity (beta-sim) of species composition and the Euclidean distances of independent variables (elevational, geographical, and environmental) for shrubs and herbs. The Mantel test and its partial version were adopted to analyze data, respectively. Linear trends of beta-sim against all ‘distances’ were obtained by ordinary least square regression. Furthermore, we employed canonical correspondence analysis (CCA) to demonstrate the relationship between species composition and the significant habitat descriptors selected through the Monte Carlo permutation test. The results showed that the turnover of herbaceous species between adjacent sampling sites exhibited a decreasing pattern along the elevational gradient, whereas the pattern of shrubs was irregular. For both shrubs and herbs, beta-sim increased significantly with both elevational and geographical distances, except the environmental distance. After controlling the influence of elevation by the partial Mantel test, the significant correlation between beta-sim and geographic distance was still maintained, but not vice versa. Results of CCA suggested that environmental factors together explained 74.4% of variance in understory species composition, and more specifically, litter thickness, slope, stem density, and basal area were indicated as key factors at the local scale. Nevertheless, there was no significant correlation between community structure and edaphic variables. We therefore concluded that environmental filtering and limitation of dispersal might jointly regulate the patterns of species beta diversity in the understory of boreal coniferous forest within Luya Mountain, with dispersal limitation playing a more important role in community assembly.