Evaluation
NIRAS has many highly experienced evaluators on staff and in our expert networks, and a long-running track record of successfully delivering evaluations of all types and sizes – including impact, process and value for money evaluation, employing a wide variety of methods. Our methodological expertise includes:
Theory-based approaches
Utilising realist evaluation, developmental evaluation and utilization-focused evaluation designs, and multiple methods, including:
+ Contribution Analysis
+ Process Tracing
+ Contribution Tracing
+ Rubrics and Scorecards
+ Qualitative Comparative Analysis
+ Outcome Harvesting
+ Most Significant Change (MSC)
+ Qualitative Impact Assessment Protocol (QuIP)
+ Theory of Change (ToC) Analysis
+ Social Network Analysis (SNA)
+ Political Economy Analysis
Experimental, quasi-experimental and quantitative methodologies
+ Randomised Control Trial (RCT)
+ Interrupted Time Series Analysis (ITS)
+ Difference-in-Differences (DiD)
+ Propensity Score Matching (PSM)
We also have experience of applying evaluation approaches at a range of different levels, from relatively small project/programme level evaluations to portfolio evaluations and meta evaluations. We have extensive experience in the management of regional and global evaluation framework contracts, including the UK FCDO Global Evaluation Monitoring Framework Agreement (GEMFA), the EU’s Framework Contract SIEA and for Sweden’s Sida since 2011. Our internal quality assurance processes ensure our services effectively meet client needs, and we also provide external evaluation quality assurance services, including for FCDO’s Evaluation, Quality Assurance and Learning Service (EQuALS).
In all our evaluations we seek to integrate Gender, Equity and Social Inclusion (GESI), including by applying an intersectionality lens in our data collection and analysis to help understand groups or individuals’ relative vulnerabilities or power
Dissemination of findings is a big part of our evaluation offer. We have extensive experience of presenting findings in engaging ways to different audiences using a variety of vehicles and tools, including presentations, webinars, blogs, infographics, videos and social media.
Check out our example "Evaluating EU support to CSOs in Nepal" by clicking the image.
A participatory, iterative and adaptive developmental evaluation of the Ford Foundation’s BUILD model
The Building Institutions and Networks (BUILD) initiative was a five-year, USD 1 billion investment in the long-term capacity and sustainability of 350 social justice and civil society organisations around the world. NIRAS conducted a developmental evaluation of the BUILD model, covering grantees in the US and Global South. With a participatory, iterative and adaptive design, the evaluation was able to flexibly respond to the changing contextual environment and the emerging findings. Employing a mixed-method approach, the evaluation placed a strong emphasis on strategic communication to build understanding, evidence, and influence.
The evaluation met both accountability and learning objectives, providing real-time feedback to inform programme adjustments, surfacing emerging issues as well as longitudinal impacts, and informed the next phase of BUILD. Inclusion and diversity are fundamental to the evaluation and NIRAS also reflected this in the selection of the evaluation team to ensure fair and balanced representation. Following the launch of BUILD 2.0 in 2021, NIRAS has been conducting a longitudinal evaluation of the new phase of the BUILD initiative (see section on our flagship MEL contracts).
A randomised control trial (RCT)-based impact evaluation of a major climate resilience programme
NIRAS partnered with the Overseas Development Institute (ODI), and two Malawian-based partners (the Centre for Development Management and Centre for Environmental Policy and Advocacy) to perform the role of Knowledge, Policy and Implementation Support Manager (KPISM) for the FCDO-funded, GBP 90.65 million, Building Resilience and Adapting to Climate Change (BRACC) programme in Malawi, which ran from 2018-2023. The primary activity completed by NIRAS was a large-scale evaluation with both quantitative and qualitative components, encompassing an impact evaluation utilising a large-scale randomised controlled experiment covering four districts.
In addition to the summative impact evaluation work, NIRAS also developed innovative fit-for-purpose developmental-based methods to inform the ongoing adaptive management of the project. The well-received final deliverables included a synthesis evaluation report, webinars and a series of briefing papers on key elements for the findings from the evaluation work.
Informing Mastercard Foundation’s work to address barriers for talented young Africans to succeed in higher education
NIRAS conducted both the mid-term and final evaluations of the initial two phases of the MasterCard Foundation Scholars Programme (MCFSP) at the University of Edinburgh between 2020 and 2023. The MCFSP’s ultimate goal is to build a thriving network of transformative leaders contributing to social and economic transformation in Africa. These were mixed-method, theory-based evaluations with a focus on identifying key emerging lessons and informing ongoing implementation of the programme. The evaluations answered the questions of what was working well and what less well – by measuring changes resulting from the intervention through quantitative methods, based on analysis of management information and conducting online surveys – why and how – examining the processes leading to these changes and identifying key contextual factors through qualitative enquiry.
During data collection a variety of channels for engagement were offered in order to ensure full as possible capturing of the perspectives of stakeholders in the Global South, and paid close attention to the extent to which there were gendered differences in the impact the programme had. Alongside the evaluation reports, NIRAS also created a suite of communications outputs to illustrate programme impact and put across key evaluation messages, including infographics, a high level summary paper, and case studies profiling individual scholars.
Evaluating Norway’s inclusion of persons with disabilities in development aid programming
NIRAS was commissioned by the Norwegian Agency for Development Cooperation (NORAD) to carry out the Evaluation of Norway’s inclusion of persons with disabilities in development cooperation, covering the period 2010-19. This was a strategic, portfolio-level evaluation covering numerous projects and countries. The evaluation focused on:
(1) the organisation of and the capability of the Norwegian development administration to meet the normative commitments related to inclusion of persons with disabilities in development cooperation;
(2) the country level results in the education sector in Nepal, Malawi, South Sudan, and Uganda; and
(3) an in-depth study of the achievements made and remaining challenges regarding inclusion of persons with disabilities in selected projects in Nepal and Malawi.
A theory-based approach was used, involving reconstruction of theories of change to determine if the underlying assumptions on how to achieve disability inclusion were valid and effective; and tailored frameworks to assess disability inclusion in the aid administration (drawing on the UN Disability Inclusion Strategy monitoring framework) and to assess progress towards inclusiveness of education systems in partner countries (drawing on the International Disability Alliance Inclusive Education criteria framework). This approach was further explored in the in-country case studies, which explored the extent to which theories of change aligned with and were achieving outcomes in actual programming in Malawi and Nepal.