Report (third draft for comments) Table of content

2.3.Selecting the evidence base

The first step consisted in collecting all evaluations of UNICEF-supported WASH programmes complying with the definition developed above, commissioned by the organisation and completed between January 2007 (one year after the launched of the current global WASH strategy) and July 2015. Reports were gathered from the online UNICEF Evaluation and Research Database⁸, from the UNICEF public website, from the UNICEF intranet based Documents Repository, from the UNICEF regional and country offices, and from past country offices’ annual reports, which include a specific section on studies, surveys and evaluations. Evaluation reports meeting the definition but that had been misnamed or misclassified in the Evaluation and Research Database as studies, survey, or reviews were reclassified as evaluations and included in the meta-analysis (n=8). Similarly, reports that had been classified as evaluations but did not meet the above definition and criteria were reclassified accordingly and excluded from the analysis (n=15). In total, 102 evaluation reports were found and taken into account for this exercise.

The second step consisted in selecting among these 102 reports the ones that would be read, analysed and used to assess UNICEF’s performance and to extract lessons learnt related to equity, scalability and sustainability in WASH programming. The following were excluded:

• 
  Evaluations of WASH programmes that were designed several years before the launch of the current global WASH strategy and whose implementation period were mainly situated before 2006 (n=3).
  
• 
  Evaluations that were rated “unsatisfactory” based on UNICEF’s independent Global Evaluation Reports Oversight System (GEROS)⁹ and corporate quality standards – and therefore not considered as a credible source of evidence. The quality of evaluations that were not GEROS-rated was assessed by the reviewers based on the same standards. (n=10)
  
• 
  Mid-term evaluations of WASH programmes for which final evaluation reports were available (n=4).
  
• 
  Full evaluation reports (or at least executive summaries) not available or available in a language other than English, French or Spanish (n=1 report in Chinese, n=1 report not available).
  
• 
  Evaluations of the UNICEF response to punctual emergencies such as a conflict, natural disaster etc. (n=19). The rationale for this exclusion is that two of the three thematic focusses of this meta-analysis (scalability and sustainability) are more associated with development programmes than with purely humanitarian actions. Mixed programming in unstable contexts and protracted crises have been included in the meta-analysis.
  

At the end of this process, 64 evaluation reports were retained.¹⁰ For the section 6 on sustainability specifically, and upon request from the WASH section in UNICEF HQ, ten additional reports have been added to this evidence base. They are the ‘WASH sustainability checks’, commissioned since 2009 by several Eastern and Southern African country offices and funded by the Government of the Netherlands. They do not always meet the criteria for defining an evaluation because, in addition to being narrower in scope, they only provide a snapshot based on one isolated data collection point and the levels of analysis tend to be more superficial than in full, typical programme evaluations. As such, they should be considered as surveys rather than evaluations. However they have been included in the analysis because of their relevance to the subject matter. See annex 10.1 for both the complete database and the final selection of reports.

Following the identification and selection of the evidence base, a quantitative description of the WASH evaluation reports was carried out using Microsoft Excel. A summary description is provided in section 3. The detailed analysis is included in annex 10.2. The 74 reports (64 evaluations + 10 sustainability checks) cover all the years between 2007 and 2015, all types of WASH intervention, and all the UNICEF regions. They cover 41 countries including all countries where WASH issues are significant in terms of absolute number of people without access to improved facilities and where UNICEF has a major WASH programme. The evidence base also comprises one global and one regional evaluation. Approximately two-thirds of the reports are final evaluations of large, comprehensive WASH programmes that were designed and implemented over a two to 11 year period. Moreover, as mentioned above, only reports rated from ‘mostly satisfactory’ to ‘outstanding’ have been included in the analysis.¹¹ Thus, the strength of the evidence base is considered good, despite some weaknesses highlighted in the subsequent sections of this report.

2.4.Analysing the content of the evidence base

All 74 reports reviewed for the meta-analysis have been read individually and in their entirety. All relevant quotes (3,025 in total) were systematically identified using a pre-established checklist. They were extracted and classified into three Excel-based matrices of analysis: one for each thematic focus. The first level of classification in all three matrices is by WASH programme component: water supply; sanitation and hygiene; WASH in schools and health centres; WASH in unstable contexts and mixed / general WASH programming. The second level of classification is the type of intervention area: low / middle-income country, and rural / urban / per-urban areas and slums. The third level is the type of finding: positive; negative; mixed / inconclusive; recommendation. The other levels of classification aim to capture the underlying conditions and factors that may have influenced the findings. They are as follows:

Equity:

• 
  The fourth level of classification for this thematic focus pertains to the stage of the programme cycle in which the WASH programme was documented to perform well (or not): identifying equity-related issues in situation analyses; targeting people in need, in particular geographically; designing and implementing the programme with an equity focus; using an equity-lensed monitoring and evaluation system; actually reducing inequities between disadvantaged and better off population groups. ¹²
  
• 
  The fifth level of classification relates to the following population groups of greatest interest: the poor and most deprived in terms of household income level and WASH conditions; women and girls; the hard to reach; elderly and disabled people; population at risk of climate change; other categories of marginalised or vulnerable population (religious and ethnic groups, people living with HIV/AIDS etc.); mixed.¹³
  

Scalability:

• 
  The fourth level of classification for this thematic focus is the likelihood of the WASH programme to be taken to scale versus the actual, observed increase in the reach of the programme.
  
• 
  The fifth one relates to the condition that enabled or prevented scaling up: existence of a scalable approach; unit costs; funding; mainstreaming; government’s willingness and leadership; institutional arrangements, partnerships and coordination; capacities at all levels; implementation guidelines and tools; engagement of the demand and supply sides of the ‘WASH market’.¹⁴
  

Sustainability:

• 
  The fourth level of classification is the likelihood / prospects for sustainability versus actual sustainability.
  
• 
  The fifth one is the category of underlying determinants mentioned in the report as a driver or obstacle to sustainability: technical determinants; financial determinants; psychological, cultural and social determinants; political and institutional determinants; quality of programme design and implementation; contextual / environmental determinants.¹⁵
  

Risk of misclassification of quote and differences in interpretation of findings across reviewers were minimised by: initial in-person training on concepts, classification categories, and their respective definitions; written checklists and instructions inserted in the matrix of analysis; peer-review provided by a second member of the team involved in the text reading and classification; and a second verification during data analysis phase. See the completed matrices in annex 10.3.

The analysis was done using simple and combined Excel filters in order to detect trends, patterns, singularities and informative cases related to each thematic focus. This was an iterative process, from the most general categories of classification (e.g. sustainability) to the most specific ones (e.g. positive findings related to the technical determinants influencing sustainability in water supply interventions), as well as by category of classification (e.g. by type of WASH intervention, by group of vulnerable population, by stage in the programme cycle, or by type of upscaling or sustainability determinant). Findings on UNICEF’s practices and performances were compared to commonly agreed standards, good practices or benchmarks. Whenever possible, performance, trends and exceptions were quantified. Cases where a weakness is likely due to a general knowledge gap in the WASH sector or intractable factors rather than a shortcoming from UNICEF are acknowledged.

The next phase of the analysis consisted of organising several brainstorming sessions with the team involved in the text reading and classification. These were instrumental for further gathering and refining the identification of trends, patterns, singularities and informative cases, for weighting the strength of the collected evaluation evidence related to each one of them, for making comparisons and interpreting the findings. This process resulted in a general and nuanced assessment of the performance of UNICEF-supported WASH programmes. In order to assess and visualise this performance, the following scoring scale and colour code was used:

• 
  Consistent evidence of good or very high performance (colour code: green)
  
• 
  Evidence of moderate or mixed performance (colour code: amber)
  
• 
  Consistent evidence of low performance (colour code: red)
  
• 
  Non-existence or unclear evidence of performance (colour code: grey)
  

This data analysis process also led to the extraction of good examples, case studies and lessons learnt. In some cases, WASH teams at country office level were contacted in order to gather additional or updated information and develop the case studies included in section in this report.