6.Sustainability of UNICEF-supported WASH programmes: findings
This section presents findings on sustainability: treatment of sustainability issues in UNICEF WASH evaluations reports; overall performance of WASH programming based on the evaluation reports; global findings by type of sustainability determinants (technical; financial; psychological, cultural and social; political and institutional; quality of programme design and implementation; contextual and environmental); and specific findings by type of WASH intervention. Each sub-section includes both positive and negative findings, as well as references to the individual evaluation reports listed in annex 10.1, and references to informative case studies presented in section 9.3.
6.1.Treatment of sustainability issues in evaluation and sustainability check reports
100% of the reports selected for this meta-analysis address the topic of sustainability to some extent (72 reports from 41 countries, plus the two global and regional evaluations).
Sustainability issues are usually addressed in a dedicated section of the evaluation reports. However, sustainability is not evaluated in a systematic way in any of the evaluation reports reviewed. Typically, the analysis of sustainability only assesses a few sustainability determinants and does not look thoroughly at all the components of the evaluated programme. For the purpose of this sustainability section, it was therefore necessary to gather information from all sections of the reports and restructure it in a more systematic way.
Sustainability checks could theoretically be used as a more valuable source of quantitative information to identify specific sustainability related issues, capture trends over time and compare between countries. In practice, they are probably useful for WASH teams in-country but cannot be used for global level analysis. Indeed, the overall quality and comparability of sustainability checks is poor because of limited sample sizes and inconsistent and generally weak indicators and data collection and analysis methods.
More than a third of the findings relate to the quality of programme design and implementation and a quarter relate to political and institutional determinants. Social, technical and financial determinants represent between 9% and 15% of findings only. Social determinants are more often discussed for sanitation and hygiene interventions while financial determinants are more discussed for water supply interventions. Contextual and environmental determinants are largely overlooked (3% of all findings).
70% of the findings pertain to the likelihood of the evaluated intervention to sustain over time, while only 30% demonstrate an actual sustainability (or lack of).
Sustainability of CLTS interventions rapidly became a major concern in evaluations and this is reflected in the number of sustainability findings, which is higher for sanitation and hygiene intervention (including mainly CLTS) than for any other type of WASH intervention.
6.2.Overall programmes’ performance
In two countries, sustainability checks have created a ‘WASH sustainability index’ (Rwanda in 2011 and 2013, and Mozambique between 2009 and 2013). It is calculated based on field level observations and surveys only. In both countries, the index ranges from 65% to 85% depending on the year. An upwards trend is observed over time in both countries. The role of the check itself in creating an incentive for a more sustainability focussed programming is unclear.
Overall, there are 36% more negative than positive findings on sustainability. Sustainability of WASH programming is a major concern in evaluations and sustainability checks.
The imbalance is particularly strong regarding sanitation and hygiene interventions (52% of negative findings, 35% of positive findings, 13% of mixed or inconclusive findings). However, evidence corroborating the widespread concern around the sustainability of CLTS interventions is limited if ODF is defined as the eradication of open defecation and the use of latrines: the average slippage rate is found around 20% one to five years after the ODF certification, which is close to the dysfunctionality rate of rural water points. When the existence and/or use of a handwashing station is integrated as a criteria for the ODF status, then the level of sustainability decreases significantly. As a matter of fact, handwashing practices are more difficult to influence, and sustain less than defecation practices – or than the use of improved water source. This is equally true in schools.
The main obstacles that constraint sustainability in WASH programming are: lack of problem analysis, feasibility studies and capacity assessments, leading to flawed theories of change and therefore incomplete or inadequate programme design and implementation; weak government leadership, engagement and capacity; insufficient focus of upstream work on government’s ownership, administrative processes and budgeting; and limited timescale of software activities (behaviour change and capacity building).
6.3.General findings by sustainability determinants
Many determinants influence the sustainability of WASH programmes and of their results. While reports do not address sustainability issues in a systematic way, all key determinants known to be associated with sustainability are addressed in at least a number of reports. Some of them have been discussed above in relation to programme scalability; they are discussed here with a different perspective. Only global findings on multi-component WASH programming are presented in this section. More detailed findings by type of WASH interventions are presented in section 6.4.
Technical determinants (17 reports from 17 countries)
One of the sustainability determinants that is usually considered as the most important in WASH interventions is the technical factor, in particular the quality of construction of WASH facilities; the assumption is that the higher the quality, the longer they will last. According to evaluation reports, quality of UNICEF WASH constructions varies a lot depending on the type of WASH intervention and the country. It is not possible to conclude from evaluations that some countries or specific interventions are generally doing better than the others. As quality of construction is neither systematically nor consistently assessed in evaluations, it is not possible either to estimate a percentage of UNICEF WASH programmes, interventions or facilities with good quality construction, or whether there is an overall improvement over time. However, sustainability checks conducted in a few countries suggest that although many technical imperfections are found in WASH facilities that make them less solid, hygienic or user-friendly, their functionality remains mostly high except for handwashing facilities in schools.
When good or poor performance is found, it is typically due to strengths or weaknesses in one or more stages of the construction project cycle:
Design of facilities and technical guidelines (Burundi 2013, DRC 2014, Timor Leste 2015, Mali 2015);
Planning of works and hydrogeological studies (Nigeria 2014);
Procurement (Indonesia 2009, Mali 2015);
Distribution of responsibilities between stakeholders and their respective capacity (DRC 2009 and 2014, Pakistan 2014, Timor Leste 2015);
Field supervision and quality control arrangements (Bangladesh 2008, South Sudan 2009, Burkina Faso 2009 and 2012, DRC 2009, 2012 and 2014, Zimbabwe 2011, Malawi 2011, Mozambique and Malawi sustainability checks 2012, Madagascar sustainability check 2014);
Beneficiaries’ involvement (DRC 2009, Mali 2015); and
Contract management (DRC 2009, Mozambique 2014, Mali 2015).
For example, issues with quality of construction material and spare parts (mainly pipes and taps) are caused by absence of detailed technical guidelines, weak capacity among construction companies and poor quality control mechanisms.
The most often cited cause for bad quality construction is by far the field supervision. In most cases, field supervision and quality control arrangements were lacking. No mechanism was in place to closely monitor the work of partner NGOs, construction companies or local service providers on a daily basis, because they were assumed to be capable. When the responsibility for supervision was given to an implementing partner – usually the government partner – it often proved unable to play its role because of limited human, material and financial capacities. The Malawi 2012 report provides an interesting illustration supported by detailed quantitative data: “The main trend observable is that about 30-40% of the 128 sites surveyed did not receive supervision during siting, drilling and development and this rose to 40-50% of the sites during yield test, civil works and installation procedures. The yield test is very important and needs a supervisor’s ‘pass’ to proceed onto civil works. With this in mind it is staggering that 43% did not receive supervision. This perhaps explains why very low yield boreholes are put into production to frustrate the users every day thereafter. The tail-off of supervision through the six construction steps is probably fund related – as the available supervisory budget is exhausted before the work is completed. The upshot of this is that 40-50% civil works are built and hand pumps installed by the contractor unsupervised – thus it not surprising that poor concrete work is common”.
The four evaluations that documented particularly good quality construction (South Sudan 2009, Malawi 2011, Mali and Timor 2015) emphasised the relevance of the full time field supervision and quality control arrangements that were put in place, and the importance of intensive capacity building targeted to all stakeholders involved in the construction works including beneficiaries. For example, in South Sudan, the initial difficulties were addressed with the deployment of UNICEF WASH specialists to three zonal offices and increased attention given to training. In Mali, the responsibility of daily field supervision was redirected from government partners to a specialised WASH engineering company, intensive efforts were invested in training both the engineering and construction companies and provide them with visuals and checklists, and the conditions of a long-term partnership with them were established. Trainings are also offered to the ministerial departments, the NGOs and the beneficiaries. The case of the Mali WASH in schools programme is interesting because after having faced quality issues in the first years of its implementation, it carried out a systematic investigation and revision of the entire construction project cycle. This initiative led to drastic improvements in the quality and sustainability of WASH facilities, as reflected in the 2015 evaluation report and sustainability check. A third party field monitor was also used (or is still used) in other countries such as DRC, where the geographic areas covered by the WASH programme are extremely large, scattered and difficult to access, and Pakistan for security and accessibility reasons.
The question of the most effective distribution of responsibilities for construction and supervision is discussed a lot in evaluation reports, but not resolved. Various configurations exist. In DRC and Timor Leste, infrastructures are built by NGOs under the supervision of the technical departments of the line ministry. This model seems to be more successful in Timor Leste than in DRC although both WASH programmes work in remote rural areas and use NGOs that have stronger technical expertise in hardware than in software activities. In Malawi, WASH facilities are built by local artisans and field supervision is the responsibility of the districts. In Burkina Faso, household latrines were built by households with backing from local masons and under the responsibility of ministerial departments at province level. In Mali, both construction and supervision are entrusted to private service providers, which are considered to be the most skilled. General oversight and punctual field monitoring is exercised by technical departments and UNICEF WASH engineers in field offices. All evaluations found that ministerial departments’ technical capacities at subnational level are very limited (Indonesia 2009, Burkina Faso 2012, Nigeria and Mozambique 2014, DRC 2009 and 2014, Mali and Timor Leste 2015 etc.). Most of their staff members have no graduate degree in civil engineering or hydrogeology, and they have little experience in construction and supervision. They are responsible for several activities in their geographical areas and have many administrative tasks to perform on a daily basis, so they are not available full time to manage or control all key steps on the construction sites. The number of staff members and their means of transportation are often insufficient to cover multiple sites at the same time and they cannot progress at the same rhythm as the construction workers onsite. Their budget is small which makes them heavily dependent on disbursement of UNICEF funds, which is reported as bureaucratic and slow in many evaluations. However, there is no agreement on who should be entrusted with the responsibility of construction and field supervision and what should be the role of the private sector. There is no golden rule. The right decision will be the one based on a thoughtful analysis of risks and of stakeholders’ capacities. The contractual arrangement shall support a realistic distribution of risks and responsibilities. The issue of who should be responsible for building latrines and handwashing facilities in schools is discussed in more details below because there are more possible options for these specific cases (see section 6.4).
The responsibility of procurement and contract management lies sometimes with the government counterpart, sometimes with the UNICEF office directly. None of the evaluations reports reviewed compare the performance of each model or discuss associated risks and mitigation measures. Only one report comments on procurement strategy, comparing the respective risks of central vs. local procurement (Indonesia 2009). It concludes that wherever good suppliers exists, procurement should be handled regionally/locally even though it may sometimes require an adaptation of the technical specifications. When risks of corruption are mitigated with appropriate control measures, local procurement reinforces local ownership and augurs better for post project repair and replacement. Another report (Zambia sustainability check 2012) comments on the bidding documents and contract specifications that are have many unclear topics, hazy descriptions and loopholes. Other procurement related issues are not addressed in the reports reviewed. Issues around bidding documents, contract and technical specifications and procurement processes are probably more common (see section on WASH in schools below) but these aspects are rarely investigated in evaluations.
Evaluation reports suggest three common mistakes in contract management (Burkina Faso 2009 and 2012, DRC 2009 and 2014, Timor Leste 2015). First, the distinction is rarely made between the technical completion of work involving the construction company and the supervisor, aiming to identify technical defects on the one hand, and on the other hand the official, substantial completion of work involving the beneficiaries, the local authorities, the ministry’s technical department and UNICEF, aiming to verify that the appropriate corrections were made, officially approve the work and ensure beneficiaries’ ownership. Second, the one year liability period following substantial completion of work, during which the entrepreneur remains responsible for any defect that appears and after which the final completion of work is organised to officially hand over the WASH facilities to the beneficiaries is not always observed. Third, poor performing implementing partners or service providers tend to be ‘tolerated’ by UNICEF in the sense that there is reluctance to break the contract or collaboration agreement in cases where it is needed.
Existence of good designs, technical standards and guidelines to be used by construction workers and supervisors has been identified as another key success factor for quality construction in many evaluations including Ethiopia 2012, Kenya 2013, Nigeria 2014, Mali and Timor Leste 2015. It may be insufficient to rely on the government’s guidance without further careful examination. It is always good practice to revise them periodically based on field experience and feedback from all stakeholders (Mali 2015).
In addition to the quality of construction, poor technical performance is also often found to be caused by post-construction management: inadequate local maintenance in communities and schools, mainly due to lack of awareness and capacity among the local management committee or operator (DRC and South Sudan 2009, Burundi 2013, etc.), and weak supply chains for repair materials and spare parts (WCARO and Ethiopia 2010, Somalia 2012, Pakistan and Nigeria 2014) (see below under ‘institutional determinants’).
Social determinants (22 reports from 20 countries)
When analysing sustainability of WASH interventions at community and household level, focussing on social determinants driving or hindering proper use and management of WASH goods and services, evaluators tend to summarise as follows: “One of the major hurdles […] is the difficulty and time required to change habits. To enable a shift in mind-set implies changes at both the individual and collective levels. It is through ongoing and sustained sensitisation efforts that a gradual shift in habits can be achieved” (Burundi 2013). “Like any major process of behavioural and institutional change, the benefits being introduced by the [WASH programme] will require years of sustained, committed effort in order to become fully sustainable” (Mozambique 2014). They conclude that sustainable results cannot be achieved within the short timeframe of the project or funding.
The literature suggests two main conditions for ‘social sustainability’: the willingness of the community and individual households (and household members) to maintain their efforts, and their ability to do so. Several underlying elements are at play.
UNICEF WASH interventions can promote motivation and commitment to social sustainability in a variety of ways. Ensuring community participation and promoting ownership and self-reliance is one of them. Participation in the design and planning of the intervention is very rare, but contribution in the implementation of field activities is very common. In general, evaluations note good engagement and support from communities in development settings in rural areas, and less in emergency settings and urban areas (South-Sudan 2009, Nepal 2009, Malawi 2011, Sudan 2012, Liberia 2012, Pakistan, Djibouti and Mozambique 2014). Very few quantitative data are available except in the Sudan 2012 report that found that “most of the participants in the focus group discussions clearly communicated their feeling that they are owners of and responsible for their water and sanitation facilities. This has been confirmed by the results of the quantitative survey, according to which over 87% of the respondents feel that the facilities of water and sanitation are the responsibility of the community: 94.2% in rural areas with WASH programme interventions, 73.9% in communities without programme interventions, and 87.7% in emergency areas). Moreover 79.3% of the respondents believe that this responsibility is actually held: 91.7% and 80.5% in rural areas with and without programme intervention respectively, 55.5% in emergency areas”. Subsidy-based approaches in water supply and sanitation face more difficulty in terms of community’s reliance on external assistance (Pakistan, Djibouti, Mozambique and DRC 2014) while CLTS interventions, which rely on community participation and resources only, generate more motivation and commitment to continue.
When community members are convinced of the risks caused by poor WASH conditions and behaviours, and aware of their own responsibilities, they are more willing to sustain the improvements created by the WASH intervention. UNICEF’s role here is to increase the level of understanding among all of them, especially decision makers and ‘agents of change’ through behaviour change communication and capacity building. These activities are part of all UNICEF WASH interventions. However, varying levels of quality and of prioritisation of these activities versus hardware interventions (construction of facilities and provision of supplies) are found across countries, as developed further below.
Involvement and leadership of community leaders is another way to stimulate motivation and commitment. Community leaders are traditional chiefs, religious leaders, ‘natural leaders’, teachers and others depending on the local context. The DRC 2009 evaluation found that in almost 60% of intervention communities, traditional chiefs took the decision of the participation in the programme on behalf of the community. The DRC 2014 evaluation found that when the traditional chief or a teacher is a member of the village WASH committee, the community action plan is implemented faster and the community is more likely to sustain its ‘healthy’ status. However, both DRC evaluations and a few other reports found that their participation in village WASH committees is too rare. The CLTS approach was successful in mainstreaming the concept of ‘natural leaders’ and using them to increase the chance of success and sustainability. In urban area, the somewhat weaker social capital and leadership can be a challenge for driving collective action and ensuring dynamism in the long run. A few evaluations noted more cases of vandalism on WASH infrastructure in populated and urban areas (DRC 2009 and 2014, Liberia 2012, Mali 2015).
Involving women is also a means to motivate communities to maintain efforts after the withdrawal of external support, as it is in their vested interest to sustain WASH improvements. As a matter of fact, they are the ones on charge of fetching water, cleaning the latrines, preparing food, buying the soap, educating children etc., so they are the ones that are the most likely to make sure that WASH benefits will last, if they are given the opportunity to do so. Their contribution to sustainability when they are part of the decision-making at community level has been demonstrated in the Bangladesh and DRC 2014 reports. Women or girl’s involvement in WASH committees is the approach adopted by UNICEF especially in rural water supply and school-based interventions, but it is too limited and constrained as highlighted in the section 4.4 above on equity and gender.
Community health promoters and children are agents of change too. UNICEF uses them as resource persons and advocate in some specific contexts and interventions only. Community health promoters are involved as facilitators in CLTS interventions in only some countries such as Ethiopia. Children are usually organised as community champions in school WASH programming primarily, with some exceptions such as Nepal and Mali where the CLTS approach is combined with SLTS and children are given more responsibilities and carry out a range of activities in their community. Significant involvement of health extension workers and children is lacking in other types of WASH interventions especially drinking water supply.
User satisfaction is an additional element that obviously affects the way beneficiaries maintain their WASH infrastructure and behaviour after the end of the UNICEF supported intervention. Ensuring acceptability, accessibility, affordability and reliability of the WASH goods and services is therefore critical for UNICEF. In general, evaluations suggest high level of user satisfaction compared to the situation before the intervention. Some concerns are expressed with reference to the functionality of water points, the affordability of drinking water for poor households, and the cultural acceptability of latrines.
Willingness and motivation at community level also depends on cultural beliefs, social norms, internal rules, incentives and sanctions. Very few ‘global WASH programme’ evaluations scrutinise these elements. Only one evaluation underlines the role of cultural beliefs, describing them as an obstacle to programme effectiveness and sustainability: “Depending on the context, which varies a lot between two provinces, two health districts, and sometimes even between two neighbouring villages, the influence of beliefs and traditions was identified as an obstacle” (DRC 2014). A few other evaluations examine the extent to which internal rules, incentives and sanctions had an effect on sustainability of specific types of WASH programming, namely drinking water supply and CLTS interventions (see further below). The lack of evaluation evidence reflects the lack of attention given by both UNICEF and evaluators to this type of sustainability determinant. However, it reflects the general lack of knowledge and evidence available in the WASH sector in general rather than an oversight from the part of UNICEF.
Besides the willingness of community members to maintain efforts in the long-term, their ability to do so is essential for ‘social sustainability’. This depends on the prevailing communication and cohesion within the community, on the knowledge and know-how available internally, and on the possible access to internal and external technical assistance.21
Little is documented in evaluation reports on internal communication and cohesion, on community organisation, ethnic and religious homogeneity, divisions and conflicts. One-third of the evaluations mention cases of vandalism and theft on water schemes and handwashing stations (DRC and Burkina Faso 2009, Malawi 2009 and 2012, Burundi and Kenya 2013, Indonesia 2014, Mali 2015). It is probably an even more widespread phenomenon. As previously noted, social dynamics and anthropological factors related to WASH sustainability remain largely unexplored so far. They seem to be considered as given facts. Nevertheless, UNICEF should try to investigate how to influence them in order to support sustainability.
Knowledge and know-how is usually considered weak in both communities and schools in spite of UNICEF’s efforts in building local capacities. Both evaluators and communities or schools themselves report that local capacities are not yet in place to properly manage and maintain WASH infrastructure without external assistance. Access to effective and continuous post-construction back up support is limited (see below under ‘institutional determinants’).
