The microbial diversity and community structure of surface sediments of Yellow River from Inner Mongolia were analyzed by high-throughput sequencing technique, and the samples were taken from Wuhai (H1), Linhe (H2), Wrad Qianqi (H3), Baotou (H4), Togtoh County (H5) and Laoniuwan (H6). The results showed that the order of bacteria abundance in six surface sediments was H3 > H6 > H2 > H4 > H5 > H1 and the order of diversity was H6 > H2 > H5 > H4 > H3 > H1, with H1 being the lowest in both cases. The three dominant bacteria groups were Proteobacteria (32.39%), Chloroflexi (13.25%) and Bacteroidetes (12.16%) in the sediments of Yellow River from Inner Mongolia through the taxonomy analysis of the sequencing data. The analysis of redundancy between bacterial community abundance and environmental factors indicated that total organic carbon (TOC), ion exchange capacity (CEC), total phosphorus (TP), total nitrogen (TN) and other environmental factors in the sediments had great influence on the distributions of bacterial communities and the negative coefficients were 82.5%, 80.1%, 85.5% and 85.2% respectively. The results of correlation analysis between microbial diversity and environmental physicochemical factors demonstrated that the pattern of microbial diversity in the Yellow River sediments was directly related to the loss of nutrients such as nitrogen and phosphorus.