Report (third draft for comments) Table of content

7.2.On WASH evaluations

General findings

Weak or incomplete evidence is generated by UNICEF through WASH programme evaluations and sustainability checks on equity, scalability and sustainability. Evaluations should be treated as a critical means of assessing the organisation’s performance, identifying relevant approaches and good and bad practices, and improving the quality of programme design and implementation in the field as well as the appropriateness of national and global WASH policies and strategies. However, this can only happen if the quality of evaluations increases and if findings feed back into management decisions and actions.

A major challenge is the methodological approach used in WASH evaluations and sustainability checks, both weak and inconsistent. Indicators, sampling strategies and data collection methods and analysis often lack robustness. Obvious bias in the data collection methods, notably over reliance on self-reports and informant interviews, and lack of triangulation are rarely acknowledged in the limitations section of the reports – when there is one in the first place. Therefore, accuracy and reliability of findings are not always ensured.

A second challenge is the weakness of M&E frameworks able to measure performance in equity, scalability and sustainability. For example, the lack of information on the quality and outcomes of IEC and capacity building activities, the lack of disaggregated result data to capture difference’s across population groups, and the lack of methodology to measure indirect results, spontaneous diffusion, or replication through upstream work. Moreover, sustainability checks have been carried out in a few countries only. In most other countries, data on sustainable outputs and outcomes are missing.

A third challenge is the absence of baseline data and of a counterfactual allowing for a comparison between before/after and with/without the intervention. As a result, evaluations are not always able to directly attribute outcome and impact-level results, especially related to behaviour change and broader health and socio-economic benefits, to the UNICEF-supported WASH programme. In other words, they are often unable to state what difference the programme really made.

A fourth challenge is the effective implementation of corporate procedures related to the quality assurance and use of evaluation findings. In some cases, evaluation reports had been misclassified as reviews, studies or surveys – or the other way around. Appropriate quality assurance processes were not applied. Even when reports had been correctly classified, they were not adequately reviewed by both WASH and evaluation experts. More quality assurance would have probably helped improve the quality and usefulness of a number of evaluation reports. Furthermore, in terms of utilisation, reading evaluations that have been conducted in the same country with a few years’ interval reveals that appropriate corrective action was not taken as a result of the first evaluation. This raises questions around the monitoring of the management response to evaluations and of their effective implementation.

More specific issues related to the treatment of equity, scalability and sustainability in evaluation (and sustainability check) reports are summarised in the below paragraphs.

Equity

Equity is addressed in most evaluation reports. This trend has increased since 2010 – which, interestingly, is also the date of the first UNICEF Gender Policy. However a definition of equity is not provided in any of the reports, and the concept is almost never unpacked. For example, evaluations do not differentiate between equity (or gender) sensitive versus equity (or gender) transformative approaches. They do not define how equity should look like in WASH programming. Moreover, evaluations gather insufficient data on the demographic, economic and social context, e.g. pockets of poverty or existence of marginalised ethnic groups, and limited information on the specific strategies aiming at reaching these population groups. These shortcomings lead to equity being incompletely and imperfectly assessed. The situation of population categories other than women and the poor is generally overlooked. The performance of WASH programmes in terms of equity is not systematically examined by following the successive phases of the programme cycle. The strength of the analysis varies depending on the programmatic phase that is looked into. In general, the level of analysis is the weakest when it comes to assess the initial situation analysis undertaken by UNICEF, the M&E system in place, and the impact effect of WASH interventions on the actual reduction of inequities.

Scalability

Performance in programme upscaling and scalability is documented in slightly more than three quarters of evaluation reports. But this analysis is not developed in a dedicated section of the reports because it is not a standard, stand-alone evaluation criterion recommended by the Organisation for Economic Co-operation and Development (OECD) for evaluating development assistance. As a result, information is often scattered between the sustainability, efficiency and relevance sections, and more rarely in the effectiveness section. Furthermore, there is a confusion between these different concepts, especially between scalability and sustainability, because evaluations do not define the notion of scalability as a basis and starting point for the analysis. Information is overall scarce. Reports generally only document cases where the programme has been replicated by other development partners or mainstreamed in national policies. Evidence of spontaneous diffusion are seldom investigated. Evaluations reviewed do not highlight as a negative finding cases in which WASH programmes have had a very limited scale and did not try to expand, as though scaling up was not an important objective of WASH interventions in development settings. Finally, the presence of the various factors that are drive or constrain programme upscaling is almost never examined in a systematic approach.

Sustainability

Sustainability is a standard OECD / DAC criterion for development evaluations and is used as such in all UNICEF WASH evaluations. It is typically addressed in a dedicated section. The roll out of sustainability checks in Sub Saharan Africa as promoted by the Government of the Netherlands is also a positive sign indicating that the sustainability discussion will be further informed by evidence. However, the same issues arise as for equity and scalability: the notion of sustainability in WASH is not defined and unpacked in any of the reports, even in the sustainability checks. For example, what sustained results would mean for WASH programming, and what the difference is between actual sustainability and likelihood for future sustainability is not made clear. After how many years can we really start to say the intervention was sustainable? What rate of water system breakdown or ODF slippage is tolerable after 1 year? After 5 years? When analysing the prospect for sustainability, what sustainability determinants should be examined? Therefore, the analysis of sustainability considers only a few number of sustainability indicators. It rarely looks thoroughly at all the components of the evaluated programme. This shortcoming is likely not specific to UNICEF WASH evaluations, however. Sustainability checks, which are specific to UNICEF, do not use common indicators and benchmarks, a well-defined sampling strategy and rigorous data collection methods. They lack baseline data. Consequently, data are often neither reliable nor comparable. They are difficult to use for global and comparative analysis.

Evaluations and sustainability checks mostly focus on the functionality of WASH facilities. The current evaluative approach to sustainability is insufficient to capture performance at service and (sub)national levels as well as performance in upstream work and sector regulation and monitoring. It also fails to examine the status of the main determinants known to be associated with sustainability, especially the environmental / contextual factors and the financial factors.