Impact Evaluation
An impact evaluation measures the effectiveness of an organisation’s intervention to change an outcome
Evaluations are critical to improving the impact of programmes and interventions. Not only do they provide insight into what has been achieved by the programme, but they also give recommendations about potential improvements. Yet, evaluations are not often popular with those involved in them. This is because evaluations need careful consideration, planning, preparation, follow-up, and analysis from those designing and implementing them and time and engagement from those being interviewed or surveyed. It can also be a long process, as evaluations mirror the length of the programmes they relate to and require work before and after the interventions.
Technical expertise required
An effective evaluation requires the skills to develop a theory of change, build logical frameworks, set performance indicators, gather and analyse data, and develop insights and recommendations. This is a daunting task for any organisation that lacks time, has low staff capacity and lacks evaluation experience. Impact evaluations bring about change, examine effectiveness, identify growth areas and ensure accountability. Organisations face multiple challenges trying to complete their impact evaluations, from effectively training staff, establishing the methodology that fits their organisation and needs, recognising the difference between correlation and causation, identifying relevant data, and creating actionable next steps. Lack of experience and practical knowledge means conducting an impact evaluation will fail to give you the knowledge you need or, worse, give you the wrong answers.
Being impartial
Even when an organisation can conduct an evaluation, it is missing one crucial skill - perspective. The work needs to be done with the willingness and openness to question current ways of working. An outside look at practices and policies, organisational dynamics, competitors, constituents, and the sector is invaluable. At Aleron, we see what you often cannot, offering a fresh perspective coupled with experience and expertise to conduct an effective evaluation. Our impartial view challenges the idea that the usual way is the best, that no other option is available, or that nothing can improve an outcome.
Our Approach
We use a structured five-step process to deliver evaluation:
-
The kick-off is an important step to ensure that we are aligned on the team roles, activities, and time-lines. We will agree with the project team on how the project management (e.g., regular catch up, reviews, sign-off meetings) and data sharing will be done during the project.
-
During step 2 we develop an understanding of the context and the desired impacts of the programme by performing research, reviewing internal documents and consulting with key stakeholders. We typically use a theory of change or logic model approach to identify the issues that the programme is addressing, the intended change / impact for the target groups and how the intervention will be delivered (e.g. activities, levers and outcomes). This exercise can be done with different degrees of engagement with stakeholders ranging from a few of workshops with the programme team to a national consultation with a broad audience over several months.
-
Based on the Theory of Change, we then design key metrics used to measure impact, as well as a comprehensive monitoring plan that sets out the data collection activities during the programme. The evaluation framework details the indicators that will be used to measure the outputs and outcomes as well as the sources for the indicators (e.g. participant feedback survey, internal activity tracker).
-
Data collection tools are designed up front based on the evaluation framework and validated with the project team. We use the Qualtrics survey platform to build and distribute surveys effectively, to ensure good user experience and high engagement. Throughout this step, it is critical to continue to meet regularly with the project team, to monitor progress (e.g., interviews scheduled and delivered, survey response rate) and mitigate any potential issues and risks.
-
In the final step, we conduct quantitative and qualitative analyses of the collected data, and collate findings in reports on an annual basis. The level of complexity of the analysis and the methods used is adapted to the datasets and the level of evidence required. When possible, our team is looking for causality between data sets and uses advanced analytic techniques (e.g. machine learning algorithm) to obtain the required insight to support our recommendations and the organisation’s decision making.
Why Aleron
At Aleron, we are passionate about measuring and improving social impact. So evaluations are a core part of what we do. We are proud to have been trusted by many organisations such as The Wellcome Trust, in2research UK, The Lloyds Bank Foundation, The Nominet Trust and The Academy of Medical Sciences to evaluate some of their programmes.
We have helped them measure and maximise their impact, improve their interventions and outcomes, expand their activities and funding, and select organisations and participants that most benefit from their programmes.
No matter what type of evaluation or programme, we evaluate your impact in a structured, professional and data-driven way. Our insights and recommendations will no doubt help you on your journey to maximise your impact and the efficiency of your operations.