The synergistic mechanisms underlying biodiversity maintenance and ecosystem functioning (productivity) and its stability in alpine grasslands on the Qinghai-Tibet Plateau represent a core scientific issue for ecological conservation in this fragile ecosystem. Based on modern coexistence theory, this study selected five representative species from four functional groups: grass (Festuca rubra), sedge (Kobresia humilis), legume (Tibetia himalaica) and forb (Potentilla anserina, Ajania tenuifolia) in an alpine grassland at Haibei, Qinghai Province. Using a three-density gradient experiment (low/medium/high: 2, 4, 8 individuals/subplot), we fitted Lotka-Volterra competition models to analyze how niche differences (ND) and fitness differences (i.e., competitive ability differences, FD) explain species coexistence. Computer simulations were further employed to investigate how ND and FD mediate diversity-productivity/stability relationships through complementary effects, selection effects, and asynchrony. The results showed that: 1) Except for Tibetia himalaica and Ajania tenuifolia (ND = -0.017), which failed to coexist due to priority effects, all other species pairs met stable coexistence conditions (ND>0 and FD≈1); 2) ND interacted with diversity to significantly enhance productivity (p < 0.001) by strengthening complementary effects while suppressing selection effects, whereas FD only influenced productivity through intensified selection effects without significant interactions with diversity; 3) ND synergized with diversity to increase community biomass mean, reduce standard deviation, and enhance asynchrony, thereby promoting stability, while FD affected stability through the diversity-asynchrony interactions. This study expands the scope of empirical research in modern coexistence theory and provides new evidence on how coexistence mechanisms determine diversity-productivity/stability relationships. The findings may offer some theoretical support for developing "key functional group configuration-diversity threshold" management strategies in biodiversity conservation on the Qinghai-Tibet Plateau.