This section presents findings on programme scalability: treatment of the issue in UNICEF WASH evaluations reports; overall performance of WASH programming based on the evaluation reports; findings on the various factors that affect programme scalability; specific findings by type of WASH intervention. Each sub-section includes a summary of both positive and negative findings, as well as references to the individual evaluation reports listed in annex 10.1, and references to informative case studies presented in section 9.2.
78% of the reports selected for this meta-analysis address the topic of scalability or actual upscaling to some extent (56 reports from 38 countries, plus the two global and regional evaluations).
Scalability is not addressed in a dedicated section of the reports. In a few reports, it is discussed as part of the sustainability section. In the other ones, it is spread across the whole evaluation report.
Two-thirds of the findings discuss the likelihood of the WASH intervention to be scaled up. Only one third of them document actual upscaling. No standards or benchmarks are established to judge whether or not a programme has reached a “significant scale”. Evaluations too rarely discuss the scale of the WASH intervention or emphasise cases where WASH interventions have not been taken to scale – as if scale and scalability were not an important objective of a WASH programme and a key concern of evaluations. Only one report emphasises the limited scale of (a sub-component of) the evaluated programme (“2 villages do not make a paradigm shift” Indonesia 2009).
Scalability issues are not evaluated in a systematic way in any of the reports reviewed except one (the global CATS evaluation 2014). Evaluations consistently fail at analysing the drivers and obstacles that will or may determine the scalability of a given intervention. For the purpose of this meta-analysis, scattered findings had to be gathered and presented in a more structured fashion that does not reflect the way scalability is addressed in evaluations.
The overwhelming majority of evaluation findings relates to organised replications and institutionalised mainstreaming efforts. Spontaneous diffusion is either not happening or not well documented.
Slightly more than 50% of findings pertain to CLTS interventions. Upscaling and scalability of WASH in schools interventions are less discussed, and of water supply interventions even less.
Actual upscaling of CLTS interventions is relatively better documented in evaluation reports than water supply and WASH in schools programming. More than 75% of findings on actual upscaling relate to CLTS interventions.
5.2.Overall programmes’ performance
There are slightly more positive than negative findings on scalability, and this trend is especially true for CLTS interventions. CLTS is the type of WASH intervention that is both the most widely scaled up and the most likely to be taken to a wider scale in the future.
Water supply and WASH in schools interventions have both a more limited scale and less chance to be rapidly scaled up in the future.
Some of the main obstacles that hamper the scalability of water supply and WASH in schools interventions are: high capital investment cost of WASH facilities that make their construction highly dependent on external funding; lack of programmatic approaches that attempt to reduce these costs while ensuring quality and sustainability; and low absorption capacity of investment.
Based on evaluative evidence, there are three ways by which WASH programming and results can spread: spontaneous diffusion within the targeted communities or in neighbouring ones without – or with very limited – external intercession; organised/encouraged replication; and institutionalised uptake at a national scale. Only a limited number of reports examine some of these pathways in detail, mostly CLTS evaluations. Only one embraces the three of them (the global CATS evaluation 2014).
Regarding spontaneous diffusion, the global CATS evaluation states that CLTS interventions “often spread outside the geographic area initially targeted via a number of diffusion mechanisms”, which are listed in the report and in other some evaluation reports as well.
In Zambia (2011), some cases were reported where “Sanitation Action Groups [village level committee] were able to trigger neighbouring villages, within close walking distance”. No evaluation reported cases in which non-beneficiary or not-yet-triggered villages changed their sanitation practices by hearing about CLTS or about the risks associated with open defecation.
Certification ceremonies are reported to spark interest by observing people coming from neighbouring villages who witness the visible results of the intervention and are stimulated by the participation of senior personalities from government and the media. The Madagascar 2014 report explains that UNICEF and implementing partners chose not to trigger all villages of the same area with the expectation that a ripple effect would occur. The evaluation states however that this approach proved too optimistic, especially because an official certification ceremony is not systematically organised in each village and non-intervention villages do not always attend.
The role of ‘natural leaders’ is also highlighted. Natural leaders are community members who emerge spontaneously during the triggering and post-triggering process as the ones who quickly understand the issue, take a resolute line and demonstrate dynamism and charisma. They typically take the lead in joining the village sanitation committee, cleaning up the community and motivating other community members. Often these natural leaders make ODF and community mobilisation their own cause and do not stop after the community achieves ODF status. They diffuse messages on good sanitation and hygiene around them within and outside their community, and sometimes carry on their efforts raising other issues related to food security, livelihood, education or gender. Natural leaders and village committees reinforce and extend the work of UNICEF’s implementing partners who also tend to communicate WASH messages beyond their area of direct intervention, for instance in neighbouring communities, or in their own family and village. Key role played by children, teachers and school health clubs is pointed out not only for CLTS interventions but also school-led total sanitation (SLTS) and WASH in schools.
Pupils are often described as ‘agents of change’ in their respective community and beyond. Consistent qualitative data suggest that they bring home the messages and behaviours learnt at school, and to a certain extent influence their family in targeted and neighbouring villages. In rural areas of most countries there is not a school in every village. So schoolchildren often do not come from the same village and can spread the word in communities that do not benefit from the WASH intervention. Rigorous quantitative data have not been found in evaluations to support this statement.
The diffusion effect of certification ceremonies, natural leaders and children remains typically limited to: increased awareness of CLTS interventions; increased awareness about the risks associated with open defecation practices; increased willingness of a community to benefit from CLTS. As pointed out in the global CATS evaluation, cases of self-triggered communities or of communities getting support from technical departments, health extension workers or natural leaders in reaching ODF status without external funding are rare if not non-existent.
Evidence of spontaneous diffusion of water supply and school WASH interventions is uncommon for the obvious reason that most technologies are inaccessible and unaffordable to communities without external assistance.
The use of mass media as an organised strategy to diffuse hygiene behaviour change has been found effective in some evaluations but not others (see section 5.5 below). Evidence of success and how to achieve sustainable results at scale is missing. Other types of encouraged or organised replication include awareness raising events in the field (school cluster meetings, inter-district meetings and jamborees gatherings mentioned in Indonesia 2014 for instance), demonstration latrines and ODF model villages (Pakistan 2014), network of natural leaders with rewards for those members having taken successful initiatives (Sierra Leone 2013), national network of community-based organisations (Mali 2015), and invitation of non-partner organisations in training events (Indonesia 2014, Mali 2015). Evaluations have attempted to measure the effect of these activities in only two countries: the experience with demonstration latrines proved ineffective in Pakistan while the case of Sierra Leone’s network is cited as a good example in the global CATS evaluation.
UNICEF has focussed on institutional uptake (mainstreaming of approaches and investment in scaling up by governments at national and subnational levels) especially for CLTS interventions. This is consistently noted as a great success of UNICEF WASH programming. Nevertheless, evaluations very rarely mention cases where WASH programmes have managed to integrate specific procedures or activities in the routine work of ministerial departments and local authorities, such as: mandatory compliance with administrative standards, rules and obligations; mandatory training; individual incentive system; benchmarking; competitions etc. Such administrative procedures and activities may have the potential to engender high impacts at an affordable cost. They might have been established in some countries with UNICEF support but have not yet been documented in evaluations.
Evaluators regret that advocacy and mainstreaming efforts often come too late in the timeline of UNICEF WASH programmes (Kenya 2009, Pakistan 2012, Indonesia 2014, Mali 2015…). While this trend is understandable because of the willingness of UNICEF and governments to only mainstream approaches that have been successfully piloted first, it is also true that the time remaining for such efforts at the end of a donor-funded programme is too short.
Performance with regards to ways of programme upscaling is summarized as follows:
