The type of litter input led to change in litter quantity and quality, which ultimately affected the nitrogen (N) mineralization in the forest soil. We chose two types of forest stand, (I) uninvaded secondary broadleaved forest and (II) the bamboo forest formed after Moso bamboo (Phyllostachys edulis) invasion, which was selected to retain, remove and replace the surface litters. The antibiotics (streptomycin and actinone) were added in the incubation experiment to analyze the relative contribution of fungi and bacteria to soil N mineralization. The results showed that after removal of litters, soil net ammonification rate increased by 27.0%, and net nitrification rate decreased by 11.4% under the broadleaved forest; moreover, net ammonification rate increased by 23.4% under Moso bamboo forest. In litter replaced treatments, the net ammonification rate increased by 43.8% and the net nitrification rate decreased by 33.5% under broadleaved forest; the net nitrification rate increased by 73.1% under bamboo forest. However, after adding antibiotics, soil fungi and bacteria played a significant role in soil net ammonification in the broadleaved forest compared with the control. Fungi played an influential role in soil nitrification under two forest stands, and bacteria played a major role in soil nitrification in the broadleaved forest. Finally, along with the chemical properties of litters, the alteration of contribution of fungi and bacteria to soil N mineralization was caused by the replacement of litters, which was directly related to lignin and cellulose content in litters. In conclusion, the removal and replacement of litters could change the rate of soil N mineralization, and the replacement of litters altered the contribution of fungi and bacteria to N mineralization in the forest soils. Our study provides a theoretical basis to elaborate underground ecological mechanism of Moso bamboo invasion into the broadleaved forest.