Do you have an impact?
When a policy is conceptualised, or a programme is established, their main purpose is often to produce some kind of impact on the socio-economic or environmental area that they are being implemented in. While policies and programmes often have many perceived impacts on an array of areas, knowing the real impact can be much more complicated. Assessing these impacts requires forethought as well as thorough planning.
What are RDP impacts?
Programme impacts are medium and/or long-term causal effects of an intervention on the programme area. The assessment of these impacts by evaluators play a vital part of the evaluations submitted in the AIRs in 2019 and ex post evaluation in 2024. Programme impacts are linked to the wider programme/policy objectives. In the case of RDPs, impacts relate specifically to the three common CAP objectives: viable food production, sustainable management of natural resources and balanced territorial development. Evaluators assess the programme´s impacts (both intended and unintended) as expressed in ‘net’ terms, which is achieved by excluding all those effects which cannot be attributed to the programme´s intervention. Net impacts include:
- The programme´s direct effects on those beneficiaries who are affected by an intervention (e.g. income, labour productivity);
- Indirect effects on the programme’s beneficiaries (e.g. leverage effect and deadweight loss);
- Other indirect effects, which go beyond the direct beneficiaries of the public intervention (i.e. which spread throughout the economy, society or environment on non-beneficiaries and other stakeholders, which have not been defined as the programme´s primary beneficiaries, such as households, retailers, and consumers).
The figure below serves to illustrate the different types of effects from rural development programmes and how these lead to net effects.
Total net impacts are usually wider than direct net effects, which only include the direct effects on the beneficiaries. The common CAP impact indicators (see figure below) provide the basis on which to assess the impacts of RDPs. In case of programme specific overall objectives, programme specific impact indicators can also be used.
Where do we begin to assess RDP impacts?
Policy achievements and impacts on rural areas are first assessed at the level of rural development programmes. For this purpose, evaluators verify to what extent the policy objectives have been achieved and whether the expected (and non-expected) impacts have materialised as a result of the programmes funded.
The impacts of a single policy intervention can, however, not be directly observed. Intervening factors from other interventions can make deciphering the attribution of specific impacts trying. The real RDP impacts are estimated when comparing the observed RDP outcomes with the outcomes which would occur in a situation without the RDP. This is called counterfactual analysis and allows one to attribute observed changes in outcomes to the programme (causality).
Evaluation approaches which include the assessment of the counterfactual situation usually require robust data and an advanced quantitative and/or qualitative assessment. Data must be accessible in a suitable format, be of sufficient quality, be in a time-series and allow the evaluator to construct the baseline for participating and non-participating groups. Generally, there are wide range of evaluation approaches and methods available to assess a RDP’s impacts. A careful review of the available data and its quality should be the starting point prior to making any decision about the evaluation approach and methods to be used.
If data for both beneficiaries and non-beneficiaries are available, then quantitative evaluation approaches are considered the ideal choice and should be strived for. However, these approaches also require specific skills and capacities needed by the evaluators and sufficient resources need to be dedicated to the evaluation if they are to be completed correctly.
If no comparison groups can be established, qualitative methods should be considered for the assessment of RDP impacts. The Method for Impact Assessment of Programmes and Projects (MAPP) is one example of a method that facilitates the collection of information from beneficiaries and non-beneficiaries needed to conduct a qualitative counterfactual analysis.
If possible, qualitative methods should not replace a quantitative assessment, but be used to complement the quantitative analysis to triangulate and further validate the findings. With qualitative methods it may be easier to gather up-to-date information and to provide a different point of view, which also explains why certain impacts occur.
Select the evaluation approaches which can produce the most robust evidence to judge realistically the RDP’s impacts and achievements to answer the common evaluation questions in the context of the AIRs or the ex post evaluation.
The latest CAP reform has shifted policy support from product to producer support and placed greater emphasis on an area-based approach. Moreover, to provide a response to economic challenges (declining rates of productivity growth, pressures on production costs due to high input prices and the deteriorating position of farmers in the food supply chain) the current CAP aims to enhance viable food production through a set of adapted CAP Pillar I and II instruments. CAP Pillar II instruments, specifically, contribute to the competitiveness of the agricultural sector operating within a properly functioning supply chain over the long term. Chapter 7 of the Annual Implementation Report to be submitted in 2019 requires Member States to show the achievements and impacts of CAP Pillar II interventions on the competitiveness of the agricultural sector in the form of net values.
What is to be assessed?
Three common impact indicators and answers to evaluation questions are related to the EU level objective of viable food production (CEQ 27).
How is this assessed?
Quantitative approaches are recommended in the Guidelines: Assessing RDP Achievements and Impacts in 2019 and can be applied for calculating and netting out the values of these three common impact indicators.
The Evaluation Helpdesk provides further guidance in the Guidelines: Assessing RDP Achievements and Impacts in 2019 on the selection of evaluation approaches for the assessment of CAP Pillar II impacts on the competitiveness of agriculture.
The European Union has long been a staunch defender of preventing dangerous climate change and has been committed both internationally and within the Union to both tackle these adverse effects as well as set an example globally. For this reason, a robust policy has been established specifically in response to environmental challenges from the increased production of GHG and ammonia emissions from intensive farming to the decline in biodiversity and countryside degradation. The CAP aims to ensure the sustainable management of natural resources and to contribute to climate actions through a number of CAP Pillar I and II instruments and Member States are required to show their achievements and impacts on the sustainable management of natural resources and climate actions throughout the programming period.
What is to be assessed?
Seven common impact indicators help to assess the EU’s policy impacts on the use of natural resources and climate actions:
How is this assessed?
In accordance with the given data-situation, very different evaluation approaches for calculating and netting out the values of the common impact indicators related to the environment may be applicable.
Two different quantitative approaches are recommended in the Guidelines: Assessing RDP Achievements and Impacts in 2019 and can be applied for calculating and netting out the values of these seven common impact indicators.
Why are CAP impacts on balanced territorial development of rural economies and communities assessed?
There are several social challenges which rural areas in the EU are currently facing, such as aging, depopulation, and the declining access to health, education and social services. A competitive and dynamic agricultural sector remains an essential driver of the rural economy in much of the EU. This is true most predominately in rural areas where the primary sector represents roughly 5% of value added and 16% of employment . Furthermore, the economic diversification of the farming sector and rural communities becomes an important factor to generate jobs and income in rural areas. This is especially relevant when it is associated with food processing, tourism, trade, and encouraging people (including the youth) to live and work there. A knowledge-based economy supported by Local Action Groups (LAGs), which leads to innovations can further support this process. The CAP aims to ensure the balanced territorial development of rural areas through the implementation of rural development programmes (RDPs). Member States are required to show the RDP’s achievements and socio-economic impacts on rural economies and communities along with the maintenance and improvement of rural employment throughout the programming period.
What is to be assessed?
Three common impact indicators are to be used in the assessment of the EU’s policy impacts on the balanced territorial development of rural economies and communities including the creation and maintenance of jobs.
To find the most suitable evaluation approach to assess the RDP’s achievements and impacts in your context please consult the Evaluation Helpdesk’s, ‘Interactive Decision Tool: Data for the assessment of RDP achievements and impacts’.
Find more information on assessing RDP impacts in the Helpdesk’s eLibrary. Find further information on Impact Indicator Fiches here.
For more information on indicators please refer to DG AGRI’s ‘CAP Interactive Dashboard’ now updated with 2017 data.
