Thematic Working Group 8 - Ex post evaluation of RDPs 2014-2020: Learning from practice

  1. Español no disponible
  2. English
  3. fr
  4. de

Background

 

In 2019, for the first time during the 2014-2020 programming period Member States had to report on their RDPs’ contributions to the achievement of the EU’s policy objectives by evaluating the policy’s impacts. The findings of these evaluations were reported  to the European Commission in the enhanced Annual Implementation Reports (AIRs) 2019.

The Evaluation Helpdesk has analysed the AIRs 2019 to identify major challenges and draw lessons for future learning. The main challenges identified through this assessment are related to:

  • The conceptual understanding of definitions of indicators and methodologies for the calculation of result and impact indicators;
  • The use of robust evaluation approaches and methods;
  • The collection and management of data for evaluations; 
  • The assessment of net contributions; 
  • The quantification of secondary contributions, synergies and LEADER contributions.

To address these challenges and to better prepare for the ex post evaluation in 2024 the Evaluation Helpdesk in collaboration with thematic experts and DG AGRI have launched the Thematic Working Group, ‘Ex post evaluation of RDPs 2014-2020: Learning from practice’ at the end of February 2020.
The objectives of the Thematic Working Group are to:

  • Address weaknesses in the assessment of RDP achievements and impacts;
  • Improve the calculation of result and impact indicators;
  • Tackle emerging issues in the assessment of priority areas (e.g. environment, climate, social indicators).

This Thematic Working Group will further serve as an important input into the future development of the monitoring and evaluation systems for the post-2020 period.

The Thematic Working Group will consist of three working packages each one related to one of the three CAP general objectives:

Each working package will encompass an overview of identified emerging issues and proposed recommendations for addressing those issues. These working packages will also be enriched with practical examples to illustrate how the identified issues have been addressed in different Member States.

To ensure that the Thematic Working Group meets the practical needs of the evaluation community ongoing consultations will take place with evaluation stakeholders through periodic Sounding Boards and other means of written feedback.
 

 

Working Package 1: Balanced territorial development

 

  • Topic 1 - Percentage/share of rural population benefiting from new or improved services/infrastructures (2014-2020): This document specifically highlights issues related to the use of the common result/target indicator ‘Percentage of rural population benefiting from new or improved services/infrastructures (ICT)’ to measure the achievements under the Focus Areas (FA) 6B (Result/Target Indicators R23/T22) and 6C (Result/Target Indicators R25/T24) in the 2014-2020 programming period. 

          twg8_working_package_1_topic_1.pdf EN PDF icon (717.65 KB)

  • Topic 2 - Jobs creation in rural areas (2014-2020): This document highlights issues related to the use of the common result/target indicator ‘Jobs created in supported projects’ to measure the achievements under the Focus Areas (FA) 6A (Result Indicator R21 and Target Indicator T20) and 6B (Result Indicator R24 and Target Indicator T23) in the 2014-2020 programming period. 

          twg8_working_package_1_topic_2.pdf EN PDF icon (853.84 KB)

  • Topic 3 - Reporting on net contributions of socio-economic impact indicators: This document examines issue related to reporting from the AIR 2019 on net contributions of socio-economic impact indicators.

          twg8_working_package_1_topic_3.pdf EN PDF icon (732.76 KB)

  • Topic 4 - Quantification of LEADER/CLLD contributions and measuring their secondary effects: This document specifically highlights issues related to the quantification of LEADER/CLLD contributions and measuring their secondary effects in the 2014-2020 programming period.

          twg8_working_package_1_topic_4.pdf EN PDF icon (837.49 KB)

Working Package 2: Ensuring the sustainable management of natural resources and               climate action

 

  • Working Document ‘Updated fiches for Complementary Result Indicators No 13, 14, 15, 18 and 19’: This document includes the updated fiches of the environment-related complementary result indicators for the period 2014-2020. These fiches have been updated by clarifying definitions of indicators and their units of measurement, updating data sources, adding additional information where useful and proposing simplifications in the methodologies.

Working Package 2: CRIs Fiches EN PDF icon (1005.71 KB)

  • List of complementary result indicators reported to European Commission in the AIRs submitted in 2019: These tables provide an overview of the reported values of complementary result indicators per Rural Development Programme, which Member States reported to the European Commission in 2019, via SFC. The indicator values were reported from the operations with primary, secondary and LEADER contributions.

CRI Values Reported in the AIR 2019 EN File (114.08 KB)

  • Working Document ‘Updated fiches for answering Common Evaluation Questions 11–14 for RDPs 2014-2020’: This document includes the updated fiches for answering Common Evaluation Questions 11 to 14 for the ex post evaluation of RDPs 2014-2020. These fiches have been updated by clarifying and updating data sources, clarifying methodologies to calculate gross and net values, proposing recommendations how to use indicators in the event of a lack of data and adding further examples and information sources.

Working Package 2: Updated CEQs 11-14 EN PDF icon (1.53 MB)

Working Package 3: Fostering the competitiveness of agriculture

 

  • Working Document ‘Best use of FADN for the assessment of RDP effects on fostering the competitiveness in agriculture’: This document analyses selected evaluation related issues when using the Farm Accountancy Data Network (FADN) data for the assessment of farm competitiveness in agriculture. 

          Working Package 3: FADN EN PDF icon (1.14 MB)

  • Working Document ‘Evaluating RDP effects on competitiveness of agriculture in a changing context’: This document addresses challenges in preparing and conducting the assessment of RDP impacts in the ex post evaluation with consideration for multiple intervening, confounding or external factors related to a changing socio-economic, environmental or health context (as recently experienced) which may influence the intended programme effects.

          Working Package 3: Context EN PDF icon (1.19 MB)

  • Working Document ‘Updated fiches for answering Common Evaluation Questions 4 and 6 for RDPs 2014-2020’: This document includes the updated fiches for answering Common Evaluation Questions 4 and 6 for the ex post evaluation of RDPs 2014-2020. These fiches have been updated by providing a non-exhaustive list of possible additional evaluation elements that Managing Authorities and evaluators may consider when preparing and conducting the assessment of competitiveness in agriculture. 

          Working Package 3: Annex 11 CEQs 4 and 6 EN PDF icon (1008.52 KB)

  • Working Document ‘Examples of evaluation elements for the assessment of RDP effects on competitiveness of agriculture’: This document provides a non-exhaustive list of possible additional evaluation elements (judgement criteria, result indicators, possible data sources) and a list of actions that Managing Authorities and evaluators may consider when preparing and conducting the assessment of competitiveness in the ex post evaluation when answering Common Evaluation Questions 4 and 6. 

          Working Package 3: Evaluation Elements EN PDF icon (942.54 KB)

Further Information

Aakvik, A. (2001): Bounding a Matching Estimator: The Case of a Norwegian Training Program. Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, 63(1), pp. 115-43.

Alliance Environnement (2018): Evaluation study of the impact of the CAP on climate change and greenhouse gas emissions. European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4.

Alliance Environnement and the Thünen Institute (2017): Evaluation study of the payment for agricultural practices beneficial for the climate and the environment. European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4.

Baldoni, E., Coderoni, S. and Esposti, R. (2017): The productivity and environment nexus with farm-level data. The Case of Carbon Footprint in Lombardy FADN farms. Bio-based and Applied Economics, 6(2), pp. 119-137.

Becker, S. and Caliendo, M. (2007): Sensitivity Analysis for Average Treatment Effects. Discussion Papers of DIW Berlin 659, DIW Berlin, German Institute for Economic Research.

Bia, M. and Mattei, A. (2007): Application of the Generalized Propensity Score. Evaluation of public contributions to Piedmont enterprises. Department of Public Policy and Public Choice – POLIS.

Caliendo, M. and Kopeing, S. (2005): Some Practical Guidance for the Implementation of Propensity Score Matching. Discussion Paper 485, DIW Berlin, IZA Bonn.

Centre for Strategy and Evaluation Services (2006): Study on Measuring Employment Effects. Brussels.

Cerulli, G. (2017): Identification and Estimation of Treatment Effects in the Presence of (Correlated) Neighborhood Interactions: Model and Stata Implementation via Ntreatreg. The Stata Journal,17(4):803-833.

Coderoni, S. and Bonati, G. (2013): Impronta Carbonica Aziende Agricole Italiane. INEA.

Coderoni, S. and Esposti, R. (2015): The evolution of agricultural GHG emissions in Italy and the role of the CAP A farm-level assessment. 29th International Conference of Agricultural Economists. Universita Degli Studi Di Milano, August 8-14.

Coderoni, S., Valli, L. and Canavari, M. (2015): Climate Change Mitigation Options in the Italian Livestock Sector. EuroChoices, 14(1), pp. 17-24. https://onlinelibrary.wiley.com/doi/abs/10.1111/1746-692X.12077.

Dabkienė, V. (2017): A Comparative Analysis of On-Farm Greenhouse Gas Emissions from Family Farms in Lithuania. Research for Rural Development, 2, pp. 225-232.

Dettmann, E., Giebler, A., Weyh, A. (2019): flexpaneldid: A Stata command for causal analysis with varying treatment time and duration. IWH Discussion Papers, No. 5/2019, Leibniz-Institut für Wirtschaftsforschung Halle (IWH), Halle (Saale).

Dillon, J.E., Hennessy, T., Buckley, C., Donnellan, T., Hanrahan, K., Moran, B. and Ryan, M. (2016): Measuring progress in agricultural sustainability to support policy-making. International Journal of Agricultural Sustainability, 14(1), pp. 31 – 44.

DiPrete, T. and Gangl, M. (2004): Assessing bias in the estimation of causal effects: Rosenbaum bounds on matching estimators and instrumental variables estimation with imperfect instruments. Sociological methodology, 34, pp.271-310;

Ecorys (2019): The Evaluation of the results and impact of the RDP 2014-2020 in the years 2014-2018: Part 3 devoted to the issue of employment and local development. Warszawa.

ENRD Contact Point (2016): Rural Development Programmes 2014-2020: Key facts & figures. FOCUS AREA 6C: Access to and quality of ICT. Brussels.

ENRD Contact Point (2019): RDPs 2014-2020: Monitoring data Rural Development Priority 6 (P6): Promote social inclusion, poverty reduction and economic development in rural areas .Situation at the end of 2018. Brussels.

European Commission (2014): Working Document. Complementary Result Indicator fiches for Pillar II. Brussels.

European Commission (2014/ rev. 2018): Guidance Document on Monitoring and Evaluation – ERDF and Cohesion Funds, Concepts and recommendations. Brussels.

European Commission (2015): Working Document. Target indicator fiches for Pillar II (Priorities 1 to 6). Brussels.

European Commission (2017): Technical Handbook On The Monitoring And Evaluation Framework Of The Common Agricultural Policy 2014 – 2020, Directorate-General For Agriculture And Rural Development. Brussels.

European Commission (2018): Impact Indicator fiches. Brussels.

European Commission (2018): Working Document for the Rural Development Committee (August 2018), Rural Development Monitoring (2014-2020) – Implementation Report Tables. Brussels.

European Commission (2019): Commission Staff Working Document, Synthesis of the findings of the evaluations of European Structural and Investment Funds Programmes, SWD (2019) 445 final. Brussels.

European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4 (2014): Capturing the success of your RDP: Guidelines for the ex post evaluation of 2007-2013 RDPs. Brussels.

European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4 (2017): Guidelines. Evaluation of LEADER/CLLD. Brussels.

European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4 (2018): Good Practice Workshop no 7.  Showing the added value of LEADER/CLLD through evaluation. Helsinki.

European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4 (2018): Guidelines. Assessing RDP achievements and impacts in 2019. Brussels.

European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4 (2019): Assessment of the progress in implementing the Evaluation Plans of RDPs 2014-2020. Chapter 2 of the AIRs submitted in 2019. Summary Report. Brussels.

European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4 (2019): Evaluation-related queries. Working Document. Brussels.

European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4 (2019): Good Practice Workshop no 12. How to demonstrate RDP achievements and impacts: lessons learned from the evaluations reported in the AIR 2019. Seville.

European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4 (2019): Interactive decision tools. Data for the assessment of RDP achievements and impacts. Brussels.

European Commission – Directorate-General for Agriculture and Rural Development – Unit C.4 (2019): Synthesis of the evaluation components of the enhanced AIRs 2019: Chapter 7. Summary Report. Brussels.

European Commission – Directorate-General for Agriculture and Rural Development – Unit E.4 (2015): Common Evaluation Questions for Rural Development Programmes 2014-2020. Working Paper. Brussels.

European Commission – Directorate-General for Agriculture and Rural Development – Unit E.4 (2016): Guidelines. Assessment of RDP results: How to prepare for reporting on evaluation in 2017. Brussels.

European Commission / DG Regio (2013): Expert evaluation network (August 2013), Job creation as an indicator of outcomes in ERDF programmes. Brussels.

European Court of Auditors (2016): Is the Commission’s system for performance measurement in relation to farmers’ incomes well designed and based on sound data? (Special Report No. 01). Luxembourg.

European Court of Auditors (2018): EU support for productive investments in businesses - greater focus on durability needed. (Special Report No. 08). Luxembourg.

Fisheries and Aquaculture Monitoring and Evaluation Support Unit & FARNET Support Unit (2018): Evaluating CLLD: Handbook for LAGs and FLAGs. Brussels.

Forschungsgruppe Agrar- und Regionalentwicklung Triesdorf / ART (2019): Bewertung des Entwicklungsprogramms für den ländlichen Raum in Bayern 2014 – 2020 (EPLR Bayern 2020), Maßnahmenspezifische Bewertung. Triesdorf.

Gelman, A. (edit) and Meng, X-L. (edit) (2004): Applied Bayesian Modeling and Causal Inference from Incomplete-Data Perspectives. John Wiley & Sons, Ltd ISBN: 0-470-09043-X.

Gooday, R. (2018): Evaluation of RDP Impacts on Emissions from Agriculture in Ireland. Presentation to the Good Practice Workshop organised by the Evaluation Helpdesk, ‘Approaches to assess environmental RDP impacts in 2019’, took place on 12-13 December, Bratislava, Slovakia.

Hirano, K. and Imbens, G. (2004): The Propensity score with continuous treatment. Missing data and Bayesian Method in Practice: Contributions by Donald Rubin Statistical Family in the book Applied Bayesian Modeling and Causal Inference from Incomplete-Data Perspectives. Edited by A. Gelman and X-L. Meng, John Wiley & Sons, Ltd.

Iglesias, A., Avis, K., Benzie, M., Fisher, P., Harley, M., Hodgson, N., Horrocks, L., Marta Moneo, M., Webb, J. (2007): Adaptation to Climate Change in the Agricultural Sector AGRI-2006-G4-05. AEA Energy & Environment and Universidad Politécnica de Madrid, Issue Number 2, December. ED05334.

Imai, K. and van Dyk, D.A. (2004): Causal inference with general treatment regimes: Generalizing the propensity score. Journal of the American Statistical Association 99:854-866.

Kelly, E., Latruffe, L., Desjeuxb, Y., Ryanc, M., Uthesd, S., Diazabakanab, A., Dillonc, E. and Finn, J. (2018): Sustainability indicators for improved assessment of the effects of agricultural policy across the EU: Is FADN the answer? Ecological Indicators, 89, pp. 903-991.

Kluve, J. et.al. (2012): Evaluating continuous training programs using the generalized propensity score. Journal of the Royal Statistical Society: Series A (Statistics in Society) 175 (2):587–617.

Leip, A., Weiss, F., Wassenaar, T., Perez, I., Fellmann, T., Loudjani, P., Tubiello, F., Grandgirard, D., Monni, S., Biala, K. (2014): Evaluation of the livestock sector's contribution to the EU greenhouse gas emissions (GGELS). European Commission, Joint Research Centre.

Louhichi, K., Espinosa, M., Ciaian, P., Perni, A., Vosough Ahmadi, B., Colen, L., and Gomez y Paloma, S. (2018): The EU-wide individual farm model for Common Agricultural Policy analysis (IFMCAP v.1: Economic Impacts of CAP Greening. EUR 28829 EN). Luxembourg: European Commission, Joint Research Centre, Publications Office of the European Union.

Lynch, J., Skirvin, D., Wilson, P. and Ramsden, S. (2018): Integrating the economic and environmental performance of agricultural systems: A demonstration using Farm Business Survey data and Farmscoper. Science of The Total Environment, 628–629, pp. 938-946.

Michalek, J. (2012): Counterfactual impact evaluation of EU rural development programmes - Propensity Score Matching methodology applied to selected EU Member States, Volume 1 – A micro-level approach. European Commission, JRC Scientific and Policy Reports, pp 1-95.

Michalek, J. (2012): Counterfactual impact evaluation of EU Rural Development Programmes - Propensity Score Matching methodology applied to selected EU Member States, Volume 2 – A regional approach. European Commission, JRC Scientific and Policy Reports, 1-83.

Michalek, J. et.al. (2020): Regional impacts of the EU Rural Development Programme: Poland’s food processing sector. 54(10):1389-1401.

Michalek, J., Ciaian, P. and Kancs, d’A. (2014): Capitalization of CAP Single Payment Scheme into Land Value: Generalized Propensity Score Evidence from the EU. Land Economics, 90:260-289.

Oenema, O., Velthof, G., Klimont, Z., Winiwarter, W. (2012): Emissions from agriculture and their control potentials. International Institute for Applied Systems Analysis, Service Contract on Monitoring and Assessment of Sectorial Implementation Actions (ENV.C.3/SER/2011/0009).

Perez Dominguez, I., Fellman, T., Weiss, F., Witzke, H. P., Barreiro Hurle, J., Himics, M., Jansson, T., Salputra, G., Leip, A. (2016): An economic assessment of GHG mitigation policy options for EU agriculture (EcAMPA 2). Joint Research Centre Science for Policy Report, EUR 27973 EN, 10.2791/843461.

Rosenbaum, P. (2002): Covariance Adjustment in Randomized Experiments and Observational Studies. Statistical Science, 17(3) 286-327.

Samson, E., van der Werf, H., Dupraz, P., Ruas, J.F., Corson, M. (2012): Estimer les impacts environnementaux des systèmes de production agricole par analyse de cycle de vie avec les données du Réseau d’information comptable agricole (RICA) français. Cahiers Agricultures, 21(4), 248-257.

Solazzo, R., Donati, M., Tomasi, L. and Arfini, F. (2016): How effective is greening policy in reducing GHG emissions from agriculture? Evidence from Italy. Science of the Total Environment 573, pp. 1115–1124.

Syp, A. and Osuch, D. (2018): Assessing Greenhouse Gas Emissions from Conventional Farms Based on the Farm Accountancy Data Network. Polish Journal of Environmental Studies, 27(3), pp. 1261–1268.

Van Doorslaer, B., Witzke, P., Huck, I., Weiss, F., Fellmann, T., Salputra, G., Jansson, T., Drabik, D., Leip, A. (2015): An economic assessment of GHG mitigation policy options for EU agriculture. Joint Research Centre Institute for Prospective Technological Studies EUR 27097 EN.

Ward, F. A. and Pulido-Velazquez, M. (2008): Water conservation in irrigation can increase water use. PNAS November 25, 2008 105 (47) 18215-18220, https://doi.org/10.1073/pnas.0805554105.

Other information sources

Complementary semi-structured interviews with evaluation stakeholders in Member States in March – April 2020

EIP-AGRI Website: EIP-AGRI, agriculture and climate change

Evaluations in Member States (e.g. Austria)

Outcomes of the focus group discussion with selected participants representing Managing Authorities, evaluators, Paying Agencies, JRC and DG Agri organised on 28 April 2020.

Outcomes of the focus group discussion with selected participants representing Managing Authorities, evaluators, Paying Agencies, JRC and DG Agri organised on 23 April 2020.

Selected Rural Development Programmes (2014-2020) and Annual Implementation Reports submitted in 2019

Suggestions and comments of evaluation stakeholders from the Member States have been collected through a written Sounding Board consultation in May 2020

Survey on data sources conducted during the preparation of the Good Practice Workshop 13

Yearly Capacity Building Events in the Member States