One plot that represents typical evergreen broad-leaved forests was established and surveyed in Tianmu Mountain National Nature Reserve, Zhejiang Province, and the plot size was 100 m × 100 m. Each tree was located in terms of x-, y-, and z-coordinates using a total station and the dominant plant populations were determined based on the dominance index analysis method. Then, V_Hegyi competition index based on Voronoi diagram was put forward, and three competition indices including Hegyi, APA and V_Hegyi were compared. Finally, the new competition index V_Hegyi was applied to analyze intraspecific and interspecific competition of dominant species. The results show that in the evergreen broadleaf forest that was studied, there are eleven dominant tree species, including Cyclobalanopsis myrsinaefolia, Cyclobalanopsis glauca, Lithocarpus brevicaudatus, Litsea coreana, Quercus fabri, Litsea auriculata, Pistacia chinensis, Zelkova schneideriana, Cunninghamia lanceolata, Liquidambar formosana and Dalbergia hupeana. The comparison and analysis imply that V_Hegyi not only overcomes the shortcoming in the inconsistency of measure units used to determine the competition based on fixed radius circles, but also can be used to make intraspecific and interspecific competition analysis and leads to stable rank of competition. Thus, V_Hegyi is superior to Hegyi and APA, and applicable for dominant tree species competition analysis. In the evergreen broadleaf forest, the intraspecific competition is fiercer than the interspecific competition. The dominant tree species whose intraspecific competition is the fiercest, Cyclobalanopsis myrsinaefolia, Cyclobalanopsis glauca and Lithocarpus brevicaudatus, also possess the cruelest interspecific competition. On the contrary, both intraspecific and interspecific competition of the tree species including Litsea coreana, Quercus fabri, Litsea auriculata, Pistacia chinensis, Zelkova schneideriana, Cunninghamia lanceolata, Liquidambar formosana and Dalbergia hupeana, are weak. Most of the dominant tree species have one main competitor, but few have more than three. Cyclobalanopsis myrsinaefolia is the absolutely dominant tree species and shows the fiercest intraspecific competition and at the same time, it is also the strongest competitor of other dominant tree species.