Ecological fragile areas are not only the most sensitive areas to the ecological environment, but also the areas where poverty alleviation issues are relatively concentrated. As an important indicator for evaluating the poverty alleviation status of farmers, poverty alleviation vulnerability can provide effective reference for the prevention of poverty alleviation of farmers in the ecological fragile areas. This paper takes Longnan Mountain, located in the West Qinling Mountains, as an example, to build an evaluation index system of farmers' vulnerability to poverty, measure the level of farmers' vulnerability to poverty by using micro survey data, and reveal its influencing factors by using the quantile regression model. The results show that: (1) the average vulnerability index of farmers returning to poverty in Longnan mountainous areas is 0.103, which is generally at the moderate level of vulnerability, and the level distribution shows an olive shaped pattern of small at both ends and large in the middle. (2) The vulnerability level of returning to poverty among different types of farmers is characterized by differentiation. High mountain areas, high proportion of agricultural income, and high dependency ratio of farmers have higher vulnerability to returning to poverty, and the proportion of highly vulnerable groups with high proportion of agricultural income and high dependency ratio of farmers is also high. (3) The difference in exposure and adaptability among farmers with different dependency ratios is the most significant, while the sensitivity level varies the most among farmers with different livelihood methods and dependency ratios. (4) The expansion of household size can significantly enhance the vulnerability of farmers to returning to poverty, while transportation convenience, network coverage, future life expectations, diversification of assistance measures, and an increase in policy assistance intensity can effectively alleviate the vulnerability of farmers to returning to poverty. The intensity of policy assistance weakens the vulnerability of highly vulnerable farmers to returning to poverty due to low transportation convenience, while the diversified assistance measures weaken the impact of high education levels on reducing household vulnerability. Finally, starting from the principle of precise policy implementation, this study proposes policy recommendations from the perspectives of the government and different types of farmers to reduce the vulnerability of farmers in ecologically vulnerable areas to poverty return and improve the quality of poverty alleviation, in order to provide broader and more accurate policy insights for many farmers in ecologically vulnerable areas in China who are facing the risk of poverty return.