Report (third draft for comments) Table of content

2.2.Defining key concepts

WASH is the acronym used by UNICEF and other development and humanitarian agencies for Water, Sanitation and Hygiene. In this meta-analysis, it encompass all types of interventions that are listed in the current WASH strategy and that UNICEF country offices are typically engaged in: Provision of community water supply through the construction and rehabilitation of water systems; promotion of good sanitation and hygiene practices including the treatment of drinking water, the elimination of open defecation and the use of latrines, and handwashing with soap (or ash); and provision of safe drinking water, sanitation and hand-washing facilities in schools and health centres. Regarding sanitation, this meta-analysis brings the community-led total sanitation (CLTS) approach into focus because it quickly became mainstreamed in most UNICEF WASH programmes after the roll out of the global WASH strategy, and this is reflected in the evaluation reports found for this period.⁴ UNICEF works for children and women but as programmes are targeted to household, school and health facility, they benefit the entire community. UNICEF WASH programming traditionally focuses on rural areas but in some countries it includes interventions in peri-urban and urban areas. In most countries, particularly in middle-income countries, UNICEF WASH programming has an upstream component aiming to strengthen the policy and enabling environment. All these dimensions are captured in this report under the ‘WASH’ acronym.

A UNICEF-supported WASH programme is defined here as an on-going or completed WASH intervention financially and technically supported by UNICEF as per the country programme signed with the government counterpart. This includes all countries where UNICEF had or has a WASH programme: approximately 100 countries during the period covered by the current global WASH strategy (2006-2015).

An evaluation of a UNICEF-supported WASH programme is defined as a systematic, objective and evidence-based effort to assess the performance of such an intervention, based on agreed criteria (e.g. typically relevance/appropriateness, effectiveness, efficiency, impact, sustainability, but other criteria are sometimes considered, including equity and gender) and benchmarks, and using various sources of data. It focusses on expected and achieved accomplishments. It provides assessments what works and why, highlights intended and unintended results, and provides strategic lessons to guide decision-makers and inform stakeholders. This definition is derived from the UNEG Norms for Evaluation in the UN System (2005) and from the UNICEF Taxonomy for Defining and Classifying Research, Evaluation and Studies (2014). In some cases this definition was insufficient to classify a report as an evaluation. For example, a report may have used only one or two evaluation criteria, or a very limited number of data sources, or did not formulate lessons learnt and recommendations. In those cases, reports were considered in this meta-analysis if they met the following discriminant criteria:

Criteria	Characteristics
Object	A proposed, on-going or completed intervention or process that has a clear timeframe and geographical scope (can be ex-ante, mid-term, end-of-programme, ex-post)
Purpose	Summative/accountability or formative/learning (for mid/end/ex-post evaluations)
Criteria	Aims to assess at least one of the following evaluation criteria: relevance, effectiveness, efficiency, sustainability, impact, and possibly other criteria such as equity/gender etc.
Data sources and sampling	Multiple sources of data are used (existing documents, management team of the intervention, implementing partners, beneficiaries, other stakeholders etc.); the sampling allows for capturing a diversity of situations and points of views rather than anecdotal evidence with limited representativeness.
Data collection methods	New data are collected from these sources of information (such as document review, survey, field observation, measurements, interviews, focus group discussions (FGDs), exercises/games etc.) and create new evidence - as opposed to only gathering and analysing data that already exist (as in a review); at least two different data collection methods are used, ideally more than two, combining both quantitative and qualitative methods.
Data analysis	Attempt to combine, analyse and triangulate/compare data in a systematic and objective manner, ideally acknowledging or addressing bias and other methodological limitations, in order to build a comprehensive, accurate/credible, nuanced and reliable picture of the evaluated intervention. End of programme evaluations ideally assess the difference made by the intervention by comparing the situation of the population before and after and/or with and without the intervention.
Conclusion & recommendations	Summarises the overall performance of the programme, what worked and why, and highlights intended and unintended results; when the purpose is formative, it formulates recommendations to inform future decision-making

There is no official, corporate definition of equity, scalability and sustainability. As mentioned above, the current UNICEF global WASH strategy and Strategic Plan propose neither a definition nor standards or good practices in these areas. Other organisations and a number of researchers have proposed definitions. However, none of them have been internationally agreed so far. For the purpose of this undertaking, a definition has been used for each of these concepts. They are based on a literature review and the author’s own judgments. They bring together characterisation elements that are commonly known and easily understood.

Equity is defined as the absence of systemic, avoidable or remediable disparities, discriminations or favouritism between population groups with different social characteristics (wealth, gender, geographical location, ethnicity, religion, age, health status, residential status, sexual orientation etc.). In the context of this specific meta-analysis, equity means that the resources, goods, services and opportunities produced by the WASH programme shall benefit the deprived, vulnerable or marginalised groups based on their needs and priorities, with the intention of reducing existing inequalities between them and more favoured groups.⁵ This definition implies that an equity lens is used at each stage of the WASH programme cycle.

Scalability refers to the ability and likelihood of a given phenomenon or action to increase its scope or size. In this meta-analysis, scalability means that the reach of the WASH programme in-country can be augmented over time compared to the initially intended reach. In the case of an actual scaling up, the evolution from a limited scale to a larger WASH intervention is able to make a difference for the country and bring it closer to internationally agreed development goals. This can happen horizontally – dissemination to new geographical areas – or vertically – uptake and adoption within the various sub-categories of the same target population. It should be noted that a commonly agreed standard for what constitutes a ‘significant scale’ for a WASH intervention does not exist, but is proposed below for the purpose of this meta-analysis. The upscaling process can be the result of a spontaneous diffusion in the field, of an intentional replication strategy developed by UNICEF, its implementing partners, the government, or other development agencies, or of an institutionalised uptake.⁶ Several factors influence the scalability of a WASH intervention.

Sustainability is the ability of prevailing national and local structures, services, processes and people to continue their role and functions after the withdrawal of all forms of assistance from the external development agency. As a result, the WASH programme outputs, outcomes and impacts would persist during a significant time period. Depending on when the evaluation was carried out along the programme timeframe (during programme implementation, at completion of the programme, or after the end of the programme), this meta-analysis looked either for the documented presence of the conditions known to be associated with WASH sustainability (likelihood / prospects for sustainability), or for evidence of WASH activities and results having been maintained during a significant period of time after donor funding has ended (actual sustainability).⁷ Again, a commonly agreed definition of what a ‘significant period of time’ means for a WASH intervention does not exist, but was set to at least one year for the purpose of this meta-analysis. A longer period of time would have significantly reduced the amount of data and evaluation evidence available.

A more detailed scoping of the equity, scalability and sustainability concepts with their respective determinants is provided in the section 2.4 describing the method used to analyse findings.