In order to improve the ecological environment of the river basin, China has implemented the pilot scheme of watershed ecological compensation, and gradually molded a watershed ecological compensation model and mechanism with Chinese distinguishing feature. At present, more than ten provinces have implemented watershed ecological compensation policies. The implementation of ecological compensation policy has played an extremely important role in improving the ecological environment of the basin, and it also had different degrees of impact on the social, economic and health aspects of the river basin. How to scientifically define and evaluate the impacts of watershed ecological compensation on the society, economy and ecological environment is of great significance to the implementation and improvement of the policy, which is also an important condition for the continuous implementation of this policy. In this study, by summarizing the evaluation methods of policy effects and combining the characteristics of watershed ecological compensation, a policy evaluation model based on Analytic Hierarchy Process (AHP)-Fuzzy Comprehensive Method and the performance evaluation index system of ecological compensation policy were established to evaluate the performance of Xin'an River Basin ecological compensation policy during the first pilot phase (2012-2014). The results showed that the implementation of the ecological compensation policy in the Xin'an River Basin was effective. However, from the perspective of promotion to the regional economic and social development, the feedback was not significant. Moreover, the determination of policy objectives, the improvement of policy system, the policy design and implementation required to be enhanced. Eventually, it is suggested to promote the establishment of a dynamic assessment mechanism for watershed ecological compensation and formulate relevant technical guidelines; intensify the infrastructure of supporting system for policy, improve the regional linkage long-term mechanism and information sharing; stimulate the objective of compensation policy transferred from ecological resources to ecological capital, improve the river's ecological flow; broaden the entrance channels for compensation funds in river basins, as well as encouraging various diversity of institution to invest into the environmental protection market. On the premise of seeking maximum ecological benefits, it should further make overall plans for economic and social benefits and make better use of economic levers for watershed governance and ecological protection to achieve a win-win situation for economy, society, and ecology. and promote the upgrading of watershed ecological compensation from the "Xin'an River Model" to the "China Model".