Defining the elements of an evaluation system
An evaluation system is composed of three core elements: an intervention logic, evaluation questions, and indicators.
/file/27433_plelements1.jpg
Intervention Logic
The programme plan or intervention logic is the logical link between the problem that needs to be tackled (or the objective that needs to be pursued) the underlying drivers of that problem, and the available policy options to address the problem or achieve the objective. The intervention logic serves as the foundation for evaluations.
Evaluation Questions
Evaluation questions define the focus of evaluations in relation to policy objectives and help to demonstrate the progress, impact, achievements, effectiveness and efficiency and relevance of the policy (e.g. ‘to what extent have RDP interventions contributed to increasing efficiency in water use by agriculture?’) Evaluation questions are the focus of evaluations.
Indicators
An indicator is a tool to measure the achievement of an objective (e.g. a resource mobilised, an output accomplished, or an effect obtained). Indicators also serve to describe the context (economic, social or environmental). The information provided by an indicator is used as a measurement tool. Indicators are aggregates of data that allow for quantification (and simplification) of a phenomenon. Indicators are the measurement tools to collect evidence for all evaluations.
Bringing all the evaluation elements together
In order to perform evaluations, these three elements must be nurtured with different evaluation approaches, methods, data and information.
An evaluation approach is a way of conducting an evaluation. It covers the conceptualisation and practical implementation of an evaluation in order to produce evidence on the effects of interventions and their achievements. Evaluation methods are families of evaluation techniques and tools that fulfil different purposes. They usually consist of procedures and protocols that ensure systemisation and consistency in the way each evaluation is undertaken. Evaluation approaches and methods help to attribute the effects and impacts to a specific intervention, thus helping policy makers understand the real value of each intervention.
Data, data, data
Data is quantitative information on selected indicators or variables which can be collected from the source itself (e.g. through survey, monitoring, statistics of entities) or secondarily through pre-existing sources (studies, aggregated statistics, etc.). Qualitative information can also be gathered to provide context for the evaluation and can be collected primarily from various stakeholders (intervention managers, beneficiaries, etc.) while using various qualitative or mixed methods (surveys, case studies, focus groups, interviews etc.). Data and information represent the evidence for the evaluation.
Consistency linkages in an evaluation system
The following figure shows the consistency linkages between the intervention logic and the other evaluation elements, evaluation questions and indicators. These must be decided prior to the start of an evaluation.
/file/27434_plelements2.jpg
The blue boxes show the relationship between the needs identified in the SWOT analysis and needs assessment (formulated as objectives), the actions (measures, activities, projects) supported with the budget that will be undertaken to achieve these objectives and the expected effects. These blue boxes make up the intervention logic. The expected effects are defined as outputs, results, and impacts:
- Outputs are actions financed and accomplished with the money allocated to an intervention. Outputs may take the form of facilities or works (e.g., building of a road, farm investment, tourist accommodation). They may also take the form of immaterial services (e.g. trainings, consultancy, information).
- Results are the direct advantages/disadvantages which beneficiaries obtain at the end of their participation in a public intervention or as soon as a facility funded through an intervention has been completed. Results can be observed when a beneficiary or operator completes an action and reports on the way in which the allocated funds were spent and managed. This can take the form in many ways, such as, it may be reported that accessibility has been improved due to the construction of a road, or that the firms which have received advice claim to be satisfied. These results should be regularly monitored. Policy makers can adapt the implementation of the intervention according to the results obtained.
- Impacts typically refer to the changes associated with a particular intervention over the longer term. Such impacts may occur over different timescales, affect different actors and be relevant at different scales (local, regional, national and EU).
The red boxes are evaluation questions, which are usually cause-effect questions asking for net effects and linked to the objectives. Evaluation questions might be further specified with judgment criteria which define how the success is to be achieved under each objective (overall and specific). They link objectives to indicators.
The green boxes are indicators, which should be consistent with the judgment criteria or evaluation questions, as well as with the expected effects (outputs, results and impacts).
Several types of indicators can be used:
- Output indicators measure activities directly realised within the programme. These activities are the first step towards realising the operational objectives of the intervention and are measured in physical or monetary units (e.g. number of training sessions organised, number of farms receiving investment support, total volume of investment).
- Result indicators measure the direct and immediate effects of the intervention. They can provide information on changes in, for example, the behaviour, capacity, or performance of direct beneficiaries and are measured in physical or monetary terms, (e.g. gross number of jobs created, successful training outcomes). Result indicators are used to answer the specific objectives related to evaluation questions.
- Impact indicators refer to the outcome of interventions beyond the immediate effects. Impact indicators are normally expressed in ‘net’ terms, which is realised through subtracting the effects that cannot be attributed to the intervention itself from the total and accounting for indirect effects (e.g. total factor productivity in agriculture, emissions from agriculture, water quality, degree of rural poverty). Impact indicators measure the programme’s effects on the area in which it is implemented. Impact indicators are used to answer the evaluation questions related to the overall objectives.
Target values are often assigned to indicators. These targets are values established based on existing information, which programmes try to realise in order to achieve the expected effects, objectives and needs.
The following figure shows an example of the consistency interlinkages between the different evaluation elements for a hypothetical case of a policy programme designed to address the problem of climate change.
/file/27435_plexample-elements.jpg
