We investigated the Da and Xia caves in a manganese mine at Songtao County to evaluate the relationship between animal community and cave habitat factors. The heavy metal, manganese (Mn), was tested to verify Mn pollution in the two caves. The animal community structures and their relationships with the cave habitat factors were analyzed and compared between the two caves. A total of 1274 specimens were collected, 660 (5 phyla, 10 classes, 21 orders, and 26 species) from the Da cave and 614 (4 phyla, 7 classes, 15 orders, and 26 species) from the Xia cave. Six communities were grouped (A-F), and their diversity indices were ranked 2.4745 (D), 1.4036 (C), 1.3549 (A), 1.1935 (B), 1.1384 (E), and 0.7265 (F). The results of Pearson correlation revealed that the content of arsenic (As) and Mn in soil correlated positively with species richness in the Da cave. Furthermore, As content was positively correlated with species dominance, and Mn content was negatively correlated with the species diversity index. Temperature was found to be the major factor for animal abundance and distribution. The results indicate that no heavy metal exceeded the moderate pollution level in the water of the two caves, except Hg, which reached high pollution levels.