In the context of climate change and rapid urban expansion, it is of great significance to understand the evolution of vegetation dynamics and its responses to environmental factors and human impacts in order to improve urban environment and support eco-city construction. However, there was still a significant lack of exploration into the spatial heterogeneity of the long-term evolution patterns and driving factors' contributions of urban vegetation since they are highly fragmented in rapidly urbanizing areas. Based on Landsat satellite remote sensing images, this study obtained the Enhanced Vegetation Index (EVI) at the pixel scale from 2000 to 2020, and took Shanghai as the research area to explore the spatiotemporal evolution and urban-rural differentiation of vegetation dynamics at the pixel level. Combining with climatic data and land use data, we proposed a framework for analyzing the impact of environmental factors (including background environmental factors and urban environmental factors) and land use change (including vegetation construction and vegetation destruction) on urban vegetation dynamics at the pixel level to quantitatively analyze how these factors affect the growth of vegetation based on multiple regression residual analysis. We found that in 2000-2020, vegetation was slightly greening in Shanghai (EVI_slope=0.009/10a) with the EVI initially decreased (2000-2007, EVI_slope=-0.071/10a) and then increased (2008-2020, EVI_slope=0.015/10a). There was a 'greening-browning-greening’ pattern from the urban core outward in Shanghai, with the urban core, suburbs and rural showed significant greening trend, while the urban expansion areas and urban fringes had shown significant browning trend. The urban core had the strongest greening trend (EVI_slope=0.053/10a) among all regions with 72% of the pixels in urban core showed significant greening. In the contrast, 31% and 33% of the pixels in urban expansion area and urban fringes had shown significant browning, respectively. The increased vegetation pixels in Shanghai were mainly distributed in the urban core and suburbs, while the decreased vegetation pixels were mainly distributed in the urban expansion area and urban fringes. Background environmental factors generally promoted vegetation greening and increased EVI by 0.033/10a, which led about 40% vegetation pixels to greening in Shanghai. Vegetation construction was the dominant factor of regional greening in urban core and suburbs by leading 42.28% and 47.83% vegetation pixels to greening, respectively. Vegetation destruction dominated the browning trend in urban expansion area and urban fringes, but environmental factors and vegetation construction could alleviate the browning trend to a certain extent The research can support planning and management of urban vegetation under global climate change and urban expansion.