Report (third draft for comments) Table of content

2.5.Contributors and timeline

The meta-analysis was conducted and the report written by WASH Evaluation Specialist Jérémie Toubkiss, under the supervision of Samuel Bickel (Senior Evaluation Advisor, UNICEF’s Evaluation Office), and in close collaboration with Cindy Kushner (Senior WASH Advisor, WASH Section, UNICEF Programme Division, New York HQ). It benefitted from critical contributions from Evaluation consultants Allison Goforth (collection of evaluation reports, quantitative analysis and textual analysis, review of the draft report), Sue Cavill, Raysa Casas and Prashant Menon (textual analysis), and Laura Olsen (co-conception of the matrix of analysis, review of the draft report). The draft report was enriched by comments from the WASH Section (Greg Keast, XXX). Quality assurance of the draft and final report was provided by the Evaluation Office (Tina Tordjman-Nebe, XXX).

The collection and quantitative analysis of the reports took place between December 2014 and July 2015. The analysis of their content and the drafting of the report was conducted between June and August. The first version of the report was finalised in September and reviewed in October and November 2015.

2.6.Limitations

This meta-analysis of WASH evaluation reports provides a useful, yet partial answer to the question of UNICEF’s performance in terms of equity, scalability and sustainability. Broader evaluations of the UNICEF response to emergencies or evaluations of UNICEF’s work in child survival and development, education, nutrition, or social protection, which may include an assessment of WASH interventions but had been classified under categories other than ‘WASH’ in the UNICEF Evaluation and Research Database, have not been taken into account in the analysis. Furthermore, the scope of this work was limited to evaluations (and sustainability checks) since 2007 and did not include other useful – and sometimes more recent – sources of information such as internal or external reviews, studies, donor reports etc. that are not consistently captured in the UNICEF Evaluation and Research Database, in the country offices’ annual reports or Documents Repository. They would help provide a more comprehensive picture of the performance of UNICEF WASH programming. Therefore, this report should be read in conjunction with other relevant sources of evidence.

The critical unit of analysis is the individual WASH programmes evaluated. That is, this meta-analysis aims to understand the extent to which each evaluation report has been able to address the above listed issues and the level of performance of each individual programme in these matters regardless of its size and of the particular time and place of its implementation. It does not determine whether variations exist across programme size, location, or over time. As a matter of fact, the number and coverage of available WASH programme evaluations across these parameters – programme size, period and location – are insufficient and therefore do not allow for sample size based meaningful conclusions. Put another way, the actual universe of reports for any of these particular sub-characteristics is too limited. For the same reason, this meta-analysis does not generalise or weight findings based on any of these factors to assess the overall performance of UNICEF WASH programmes globally, nor does it compare performance across the various evaluated interventions due to varying contexts, time spans for implementation, programme maturity etc.

Findings are based on the content of evaluation reports. It was possible neither to access primary data to verify accuracy of analysis in the evaluations, nor to contact programme managers to gather missing information, collect additional data, and correct evaluators’ potential misperceptions. It is possible that some reports, even if rated as mostly or highly satisfactory, do not depict an exact and comprehensive picture of the programme and to some extent reflect the position put forward by the WASH team being evaluated or the evaluators themselves. Nevertheless, because this meta-analysis report looks at global findings, patterns and trends across all evaluations, weaknesses in individual evaluation reports are reduced. This also has the associated effect of smoothing out variations and nuances; as in any meta-analysis exercise, unique cases of findings may be lost. To counter-balance this, a dedicated section at the end of the report highlights examples of success and good practices identified in evaluation reports.

Since the level of rigour of data collection methods and analysis varies a lot across reports (especially among those rated mostly satisfactory) and, in some cases, could not be assessed, the meta-analysis mainly uses data and findings whose accuracy, credibility and consistency are estimated sufficient. Whenever necessary, uncertainty or weaknesses in the quality of evaluation evidence is explicitly acknowledged. Besides, specific findings from one report are compared with those of the other reports. Again, the fact that this meta-analysis looks at the global evaluation evidence and does not draw any conclusion based on a single report mitigate the risk of including evaluation evidence whose quality is unclear or potentially weak.

Lastly, because the frequency of country-level evaluations is variable, some of the country-specific findings presented in this report are based on evaluations that date back from several years and may not be valid anymore. That is why the year of the referenced reports is systematically specified.

3.Overview and quantitative analysis of WASH evaluation report

This section provides a short quantitative description of the universe of WASH evaluation reports since 2007, including those that were excluded from this meta-analysis. Additional details and graphs are to be found in annex 10.2.

3.1.Coverage of evaluation reports

The 102 UNICEF WASH evaluations cover all years since 2007. The average number of WASH evaluations per year is 13.5 (excluding the current year 2015). This average is the same for reports included in this meta-analysis when including sustainability checks.

Evaluations also cover all UNICEF intervention regions. Evaluations from Eastern and Southern Africa (ESA) are the most numerous, followed by the ones from the Western and Central Africa (WCA). Both regions combined make up more than half of the WASH reports overall and of those included in this meta-analysis. They are followed by the Eastern Asia and the Pacific (EAP) and the Middle East and Northern Africa (MENA) regions, both 12% of all WASH evaluations, and by South Asia (SA, 10%). The Latin America and the Caribbean region (LAC), Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS) and UNICEF Headquarters (HQ) make up 10% of the 102 WASH evaluations all together. The order of regions is the same as for the reports included in the meta-analysis. Overall, the quantitative analysis revealed that the number of WASH evaluations in WCA, MENA and LAC is slightly insufficient when compared with the share of these regions in the global UNICEF WASH expenditures.

Country-led WASH evaluations and sustainability checks have been conducted in 52 countries, that is to say only a little more than half of the countries where UNICEF engages in WASH. 41 countries are covered by the meta-analysis, including the top 15 countries with the highest WASH expenditures (regular/development and mixed programming) since 2007: the Democratic Republic of Congo (DRC), Ethiopia, Nigeria, Bangladesh, Pakistan, Zimbabwe, Kenya, Sudan, Mozambique, Zambia, Somalia, India, Mali, Sierra Leone and the Philippines. Almost all of these evaluations are programme or project evaluations. Very few of them are joint or impact evaluations (8 and 4 reports respectively). In addition, one regional evaluation led by the WCA regional office and one global evaluation led by HQ are part of the meta-analysis.

Regarding the range of evaluation topics, 68% of all evaluations conducted since 2007 pertain to regular programming, 22% emergency programming and the remaining 10% mixed programming. In terms of type of WASH intervention, 23% of them address rural WASH programmes in general, 22% WASH in schools, 17% CLTS and 11% hygiene promotion. The remaining 27% are distributed between drinking water supply, urban WASH, upstream work / institutional strengthening, and specific emergency interventions. It is to be noted that an individual evaluation can address several topics and that none of the reports address the latter topics exclusively. This distribution of topics is similar among reports included in the meta-analysis with one exception: As explained above, purely humanitarian evaluations were excluded from the scope of the meta-analysis. Aside from that, they cover all types of WASH intervention.