Financial determinants (26 reports from 19 countries)
All evaluations emphasise the need for continuous financial flows beyond the duration of the UNICEF-supported intervention. They must come from all possible sources: the beneficiaries, the government and local authorities, and development partners. The three main forms of finance for the WASH sector are known as the ‘3 Ts’: tariffs (user fees and contributions), taxes (funds raised by national/regional/local governments and typically channelled into the WASH sector in the form of subsidies or loans), and transfers (payments coming from foreign sources, official development assistance and private philanthropic contributions). Any WASH programming and the WASH sector in general should strike the right balance between them. Unfortunately, WASH evaluations examine neither the three sources of finance nor the achievement of the right combination of them in a systematic manner when analysing the financial factors related to sustainability. Thus, the following paragraphs are based on a compilation from several partial analyses of financial sustainability. They are not meant to compare UNICEF’s practices with a ‘gold standard’
As mentioned earlier, the participation of beneficiaries is said to contribute to individual and community ownership and therefore to sustainability. This participation often takes the form of a financial or in-kind contribution to the capital investment cost or of an upfront contribution to an operation and maintenance fund. The absence of such a contribution partly explains the lower level of ownership in emergency areas according to the South Sudan 2009 evaluation. On the contrary, the extent of the households’ contribution is believed to be a strong asset in the sustainability of CLTS interventions.
Most evaluations rely on self-reporting to estimate the ability of beneficiaries to pay for the water tariff, for the maintenance and upgrading of latrines, and for hygiene supplies, which is not the most reliable method. They conclude that their ability to pay is very low given the income level in rural and peri-urban areas where UNICEF operates, which does not bode well for future sustainability. This finding is supported by very limited quantitative data. Only the Liberia 2013 report attempted to estimate the financial viability of the water service and the public latrines installed by UNICEF in the slums of Monrovia: “The outcomes of such calculations suggest that nearly all water points and public latrines are financially viable and sustainable beyond the short to medium term”. This finding suggests that a distinction needs to be made between beneficiaries’ ability and willingness to pay. UNICEF’s efforts and results in supporting beneficiaries’ ability and willingness to pay through awareness raising, capacity building and pro-poor financial instruments are discussed in other sections.
As stated in the Mozambique 2014 evaluation, there is a need for continued funding from governments to support the national WASH sector, develop in-country capacities and strengthen WASH sustainability at service level when needed: “Only government […] can assure that this dimension of sustainable capacity is achieved through future recurrent budgetary provision. Fiscal capacity of users and community institutions is modest.” Evaluations found that the financial contribution from national governments is generally uneven and insufficient. It is higher in middle-income countries (Nigeria, Sudan, Bolivia, Indonesia, Ghana and Pakistan are documented examples) than in low-income countries. At subnational level, the devolution of power is reported to have worsened the situation as adequate financial resources have not been transferred from the central level, and local authorities’ ability to leverage local resources through taxes is limited. UNICEF WASH programmes need to step up their advocacy efforts in this area. Other tax-based approaches can be explored such as the ‘polluters and users pay’ principle enshrined in national Water Code (as in DRC), national solidarity funds for WASH financed by a surcharge on the water bill, or other financing mechanisms.
While it is clear that the ultimate goal is to develop national WASH sectors that are able to rely on national resources only as opposed to time-bound and unstable donor funding, it is also obvious that it is a long-term ambition in most countries where UNICEF is present. In the meantime, international transfers will remain necessary. Continuous funding from UNICEF and other donors is needed to ensure a minimal level of sustainability. UNICEF’s fund raising efforts are constrained by WASH donors’ capacity and priorities. But WASH sections at all levels of the organisation are expected to design ‘bankable programmes’, build a relationship of trust with their donors and diversify their funding sources. Various levels of success have been found regardless of the type of WASH interventions (see section 5 above). The Kenya 2013 report exemplifies some of the shortcomings in this area: “There is scope for further diversification of partnerships in order to avoid any perception of ‘donor capture’ of the programme. UNICEF needs to work with other bilateral partners to map out a more effective resource mobilisation strategy which recognises the importance of sustaining investment in rural WASH within deprived areas of the country. This must include engagement with small scale private sector service providers and should seek to open up access to rural credit finance – possibly through the rural banking sector. Scandinavian donors (Sweden, Denmark and Norway) retain strong interest in a ‘balanced’ public-private partnership approach in the sector and DfID of the UK have recently published a technical position paper promoting small scale private sector partner mobilisation in Africa which would suggest that they too could become willing investment partners.” The WASH SWAp was officially launched in 2006 culminating in the signing of the Joint Cooperation Agreement. The Agreement was signed between the Ministry of Water Resources and Irrigation, DANIDA, GTZ, SIDA, UNICEF and was to incorporate new partners on a rolling basis. The evaluation states that this was “a move to enhance stakeholder policy alignment, mutual accountability, harmonise implementation of the sector reform process, undertake joint planning and monitoring and leverage further investment to the sector. UNICEF has been singled out for particular criticism for demanding separate and substantive accounting procedures, though we understand that this was a condition imposed by both the principal donor and UNICEF Headquarters (Comptroller Division) rather than by a local programming decision”. Nonetheless, “a UNICEF WASH staff should attend all national level sector steering committee meetings and key technical working groups, specifically when strategic resource mobilisation and allocation to the rural sub-sector issues are discussed.” Furthermore, “The Government of the Netherlands remains committed to working within the Kenyan rural WASH sector, but need to be re-assured that UNICEF is capable of delivering the level of technical and administrative support to achieve remaining ambitious sector targets. UNICEF WASH’s response should include a frank re-assessment of bottlenecks […]. Donor confidence is paramount to the future of the WASH programme and for reasons already mentioned, a more frank and compliant (with donor agreement provisions) account of progress will be required in order to maintain confidence and investment support to the programme”.
UNICEF’s own donors have an important role to play not only in terms of amount of funding but also in terms of how they disburse their aid. The need for long-term, predictable donor commitment comes up in a number of reports. Many of the institutional changes and capacity building efforts can only materialise in a longer horizon than the few year timeframe typically imposed by donor funding. Furthermore, funding arrangements are key. Channelling and pooling funds through SWAp or basket funds has been identified as a practice that is beneficial to sustainability. The Malawi 2011 report esteems that “the policy of direct funding at district council level has created strong ownership and competence at decentralised level” and thus supported sustainability. Accompanying foreign aid with sustainability-focussed conditionalities such as the ‘sustainability compacts’ promoted by the Government of the Netherlands in the ESA and WCA regions, and the ‘pacte de performance’ (performance pact) in DRC can also be cited as interesting initiatives likely to push sustainability issues up in national governments’ agenda. Regarding the sustainability compact and related sustainability checks, the Kenya 2013 report asserts that “the Government of Netherlands support to the WASH programme has had a significant and beneficial impact upon the programme’s address of WASH Sustainability issues. Re-focussing on the sustainability of WASH service gains becomes critical to maintaining resource mobilisation momentum for the sector among both the Government of Kenya and other bilateral partners. Important recent work undertaken in country and at Headquarters level in developing Sustainability Compact models and in building capacity among sector professionals to undertake Level 3 monitoring of bottlenecks to sustainability will be of great value in strengthening performance in this area”, while the Mozambique 2014 report affirms that “sustainability checks have led to actions to improve sustainability notably for water supply”.
Political and institutional determinants, sometimes referred to as the ‘enabling environment’, encompass a range of conditions both at national and subnational level. Many of them are also key factors for scalability:
Existence of sound national policies and strategies;
Ownership, willingness and leadership of national governments;
Institutional arrangements, partnerships and sector coordination;
Capacities at all institutional levels;
Existence, dissemination and use of implementation guidelines and tools; and
Engagement and capacity of the service providers.
An additional one is associated with sustainability: capacity at community level.
National policies and strategies act as a ‘sector roadmap’. They enable sustainability for several reasons.
They set up a supportive legal and regulatory framework for operation and management.
They are meant to remain stable in the medium term.
They define implementation strategies and approaches to be followed by all sector stakeholders, reducing potential inconsistencies or incompatibilities that often jeopardise results in the short to medium term.
They impose rules and disseminate good practices.
They clarify the distribution of roles and responsibilities between ministerial departments, other national and local authorities, development partners, civil society organisations and the private sector.
When accompanied with inclusive and dynamic coordination mechanisms, they reduce possible conflicts or mismatch.
For all these reasons, UNICEF WASH programmes must engage in upstream and advocacy work aiming to reinforce the WASH enabling environment. In a few countries, evaluations have found too little engagement from UNICEF (Palestine 2012, Kenya 2013, Pakistan 2014). As mentioned above, evaluations document tremendous investment from UNICEF in many countries, both in the policy environment and sector coordination platforms (South Sudan and Ghana 2009, DRC 2009 and 2012, Zimbabwe and Bolivia 2011, Sudan 2012, Timor Leste and Mali 2015 etc.). The level of success is generally good, except in very unstable countries and in countries with little government leadership (Mali and Timor Leste 2015 etc.) where progress is slower and policies are not enforced in a consistent manner. As the Timor Leste evaluation states, “there is a tendency amongst some government officials to insist on more involvement and a higher profile, but in practice to take a back seat arguing a lack of resources”.
The level of ownership and leadership from government counterparts at national and subnational levels is probably a more common issue than what is reported in evaluation reports, because evaluators tend to not investigate it when assessing the prospect for sustainability. It is, however, a prerequisite as government institutions are then more likely to take up their responsibility in post-implementation follow up support and regulation (monitoring, incentives and sanctions) as well as in terms of financial commitment. While political willingness, leadership and ownership often lies beyond UNICEF’s influence, the analysis of evaluation reports helps identify three ways of supporting them: programme management and coordination arrangements; targeted and continuous advocacy efforts and the use of ‘WASH champions’; and effective capacity building of public institutions.
Three different models of programme management and coordination arrangements have been found in the reviewed reports. The first model is one in which UNICEF is responsible for programme implementation and operates in the field mainly through civil society organisations (CSOs) and private service providers. Selection of partners in charge of software and hardware activities, procurement, contract management, and most financial disbursements and reporting are directly managed by UNICEF. The role of the government is to participate in some key activities such as training of community based WASH committees and teachers, punctual field monitoring and validation of key outputs (completion of construction works, institutionalisation of water management committees, ODF certification etc.), and in coordination mechanisms at national and subnational levels. This model was found in Burundi, Mali, Mozambique, Pakistan etc. It reflects the prioritisation of effective and efficient delivery over government ownership and sustainability. The second model is opposite, in which UNICEF acts more as a donor, advocate and supporter to the government than as a direct implementer. The programme is co-designed by the government partners. They carry out most of the field activities, with UNICEF being sometimes responsible for directly managing some other implementing partners and specific activities including significant upstream work. This model was found in Nigeria, Ethiopia, DRC, Ghana, Zimbabwe, South Sudan, Bosnia Herzegovina etc. The third, intermediate model was found for example in Indonesia, Philippines and Sudan, with decision-making, implementation and coordination being largely decentralised but only partially embedded in government bodies. The second model is obviously the one that encourages ownership and leadership the most. It is also the most challenging one given the risks related to weak implementation and management capacities of government partners as well as poor fund management and reporting. It requires more training and control measures as well as strong communication and partnership expertise. It is interesting to note that the choice between the three models does seem to be based neither on the income and development level of the country, nor on the level of government management and implementation capacity, nor on a risk assessment and the search for the better trade-off. It more probably reflects the management culture in the UNICEF office or in the WASH section, or a deliberate decision by the WASH chief.
As shown above particularly with reference to CLTS, advocacy efforts can go a long way in creating governments’ ownership and leadership if they are structured, targeted and continuous (as exemplified by the Ghana Guinea worm eradication programme). Collaborating with other sections within UNICEF (Social Policy, Education, Health etc.) and partnering with other external and national development partners and networks are also good practices, yet UNICEF’s engagement in this direction is rarely testified in evaluation reports. Identifying and arousing WASH champions within governments’ institutions also helps support external advocacy efforts (Indonesia 2009, WCARO 2010, and India 2013).
Capacity of government’s institutions at central and subnational levels is a driving factor for sustainability and they are found to be insufficient in all evaluations as mentioned above – even in middle-income countries. Obviously, if governments’ capacities were strong, UNICEF’s presence in country would not be needed. Moreover, central governments’ investment in their technical services at regional level and in local authorities is weak, maybe because of a reluctance to share resources and power. The challenge is generally not underestimated: The majority of evaluations found that UNICEF WASH programmes have invested significant time and resources to develop guidelines and training modules, and facilitate workshops. For example, the Sudan 2012 report calculated that 10% of programme budget was spent on capacity building – against 1% in Pakistan. Besides Pakistan, only a few other WASH programmes have been found to make insufficient (or untimely) investment compared to the needs: DRC, Burundi, Kenya and Indonesia. However, all evaluations found that UNICEF’s engagement in capacity building is suboptimal, because the challenge is inadequately addressed. Some programmes have invested in training sessions without having developed quality training materials, post training reference materials, and guidance document and tools for practical implementation (this is the case reported for example in the Sudan 2012 evaluation) – or the other way around. Others have trained government and implementing partners at national and regional levels, but the training of trainers approach proved ineffective in reaching down to the local and community level (Nigeria and Pakistan, for instance). The method used is usually a few day workshop in a meeting room. Hands-on training, on-the-job support and mentoring are uncommon. Topical focus of training materials is generally adapted to the target audience but do not cover all the capacity building needs. Trainings are delivered as one-shot initiatives and it is too often hoped that they will create lasting benefits without a post-training mentoring period and without refresher sessions. The only exception is probably in CLTS programmes where more emphasis is put on practical training and learning by doing. The Ethiopia 2010 evaluation noted that “in spite of the fact that training and capacity building is a very high cost programme component, no periodic evaluations were undertaken to assess effectiveness of training conducted for different target groups”. This remark is true for almost all other countries and all types of WASH intervention.
Only two programmes are reported to have rolled out quality training and achieved good results: Bosnia Herzegovina and Zimbabwe. Both adopted a systematic approach to capacity building based on a needs assessment. They developed a capacity building plan accordingly and compared several training methods to find the most suitable ones. They partnered with an institution that provided training services when UNICEF or government partners’ lacked the required competence. The trainings aimed not only to raise awareness and knowledge on WASH issues and programme implementation approaches, but also to reinforce organisational and management skills that have been neglected in almost all other countries such as: planning, monitoring and evaluation; procurement and contract management; financial management and reporting; process documentation and learning; IT skills; and community mobilisation and development technics. It is to be noted that the mid-term evaluation of the WASH programme in Bosnia-Herzegovina specifically assessed the capacity building component and formulated concrete recommendations that were subsequently implemented by UNICEF. This generated the good results observed in the final evaluation, confirming the importance of ex-post assessment of training activities as recommended in the Ethiopia 2010 report.
The intensity, quality and relevance of capacity building activities can be high without resulting in the expected impact. Effective and sustainable results also depend on the existence and proper identification of trainable staff, whether interest can be created among them and whether they remain in place. Staff turnover is particularly problematic for sustainability, as evidenced in all evaluations reports (and in the section on scalability above). Staff members who have been trained and have gained know-how and practical experience are regularly transferred to other geographical areas and departments, or retire. Others, typically the most qualified, leave for better paying jobs in better resourced and better managed external support agencies or private sector companies. In some emergency countries like Zimbabwe or South Sudan, they even relocate abroad in search of a better living environment. It affects knowledge retention, leaving UNICEF and implementing partners’ staff to regularly work with new officials with no previous exposure to the programme or even to the WASH industry. Institutional memory and investments in personal relationships are lost. This phenomenon, largely beyond UNICEF’s control, undermines capacity building efforts and can discourage active involvement of government officials, civil servants and local authorities. Many of the benefits of the project will fade away as a result.
Capacity building activities are typically financed by international aid, and those supported by UNICEF are no exceptions. The relatively short timeframe of most programmes and the limited budget available inevitably reduce opportunities for long standing, far reaching and sustainable capacity development. As highlighted in the Mozambique report from 2014, UNICEF “cannot ‘build capacity’ over a couple of years and consider the job done. Refresher training (partly, but not only for newly appointed officials [and civil servants]) and advisory support will be needed into the indefinite future. Only after a decade of such support can there be more confidence that capacity will be sustained.” The ideal strategy, therefore, is to formulate a long-term capacity building plan, organise continuous training sessions through in-country resource institutions, and embed these on-going activities in government’s administrative procedures and budget. This requires strong advocacy efforts and partnerships with local universities and vocational training institutions. Some WASH programmes have had some success with teacher and pupil training on WASH because teacher training and the school curriculum are already well structured processes managed and financed by national governments, on which UNICEF can piggyback more easily. There is a similar, yet unrealised potential for integrating WASH in the vocational training of health care workers, but such an opportunity does not exist in other parts of public administration.
In addition to training, most UNICEF WASH programmes provided vehicles, motorbikes, computers and other office equipment to government partners. Evaluators value such material support, as without it implementation, field monitoring and follow-up support would be impossible (Ghana 2009 and 2012, DRC 2009 and 2014, Mozambique 2014 etc.). Nevertheless, it poses a considerable problem for UNICEF not only in terms of cost, but also in terms of control of adequate dispatching and appropriate use of this material. Its maintenance is the responsibility of the government, and depends on available operating budgets. It is clear that the need for material support can quickly become endless. In Mali, Zimbabwe and Indonesia, the WASH section chose not to provide this type of support and lead to issues in terms of aid fungibility. It is surprising that none of the evaluators examine the pros and cons of this practice, and discuss whether the role of UNICEF is to equip government institutions or to advocate for that to the government. None of the reports reviewed mention an agreement that clarifies the terms of external assistance expected from development partners. It is not clear where a government’s responsibility ends and where the contribution of development partners begins. This issue goes, of course, well beyond the specific case of UNICEF and WASH development programmes.
Private providers of WASH goods and service will often be the ones responsible for managing water schemes, repairing and upgrading water and sanitation facilities, providing spare parts and hygiene supplies etc. after the withdrawal of external support agencies. They significantly contribute to the sustainability of existing WASH services. Their availability, capacity and engagement are constrained in rural areas. UNICEF has attempted to nurture a range of private entrepreneurs through contractual agreements and training: Local latrine artisans in DRC and Malawi; latrine builders and pump mechanics in Zimbabwe; latrine artisans, drilling contractors, hand dug well and spring developers, spare parts suppliers and pump mechanics in Ethiopia; soap producers and water treatment technologies entrepreneurs in Nepal; WaterGuard suppliers in Liberia (trained on social marketing); civil engineering and drilling companies as well as construction supervisors in Mali etc. Pakistan and other countries engaged with a consulting company or NGO to design low cost latrines that could be marketed before or after a community becomes ODF. In Mozambique, UNICEF trained local artisans and supported private spare parts suppliers as well as maintenance service providers. However, UNICEF’s expertise is still developing in terms of how to engage with and develop capacities of private entrepreneurs, and to establish viable business models in order to make real difference for the whole national WASH sector. As the Kenya 2013 report highlights, there has been a marginal attention given to and even an “historical mistrust” of private sector interests in rural WASH until the late 2000s. There is a lack of commitment from UNICEF in “strengthening business regulation systems, assisting with the strengthening and/or formation of professional associations, raising standards of professionalism, developing best practice guidance, promoting self-regulation, […] clustering private operators and building innovative and sustainable business models for WASH services”, including “alternate sources of financing for small scale private sector service providers such as the extension of insurance services to guarantee funding and service response to rural communities when [WASH] systems need repair”.
Ultimately, the local authorities and communities are in the front line of post-construction management and maintenance. Their capacity is even more of a challenge, especially in Sub-Saharan Africa where the literary rate and financial capacity are low. Again, UNICEF tries to address this challenge through beneficiaries’ organisation and training, with the objective of creating ownership, empowerment and self-reliance.
Beneficiaries’ organisation typically means setting up new local commissions as well as community and school-based committees specifically dedicated to WASH. Evaluations identify five common mistakes.
Too many committees are created, e.g. a communal WASH commission, a community water management committee, a village sanitation committee, a school health club etc. Focussing on the community level, the South Sudan 2009 report stresses: “Under this project, a number of committees have been established at community level. They include Village Health Committees, Village Water and Sanitation Teams, and Village Water and Sanitation Committees. The sustainability of the services introduced under this project is very largely dependent upon the sustainability of these committees themselves.” Yet, these committees all require committed members and resources that maybe be lacking. Conflict around distribution of roles and responsibilities may also arise. There is a need for rationalisation.
These committees come in addition to what already exist instead of being integrated on the prevailing community structure. In particular, traditional community leaders, women’s and youth groups, village saving and loan associations, teachers and schools clusters, and community health workers are often overlooked while piggybacking, integrating and mainstreaming WASH in their respective agenda would sometimes be more effective and efficient.
Committees tend to be created too late in the activity cycle – typically during or after the construction phase – which reduce the level of buy-in.
Most committees rely on the voluntary work of their members. Unpaid labour weakens motivation and accelerates turnover. It is therefore detrimental to the functionality and sustainability of WASH committees.
Not enough attention is given to internal and external processes that are meant to ensure sustainability of local committees after the end of the project. The South Sudan evaluation names them as the ‘4 Rs’: “The first ‘R’ denotes the existence of mutually agreed Rules and norms governing behaviour of community members. The second ‘R’ represents the establishment of a [Regulation and] Referee system for [monitoring and] checking compliance with the rules. The third ‘R’ represents a Reward and sanctions system for incentivising and providing benefits to those who comply with the rules or for punishing those who fail to comply. The final ‘R’ represents a system for Resolution of conflicts and addressing grievances. If any of the first four Rs does not exist, perverse incentives set in, and the glue that binds communities together begins to crumble”. Most of the four Rs are generally found to be missing in the various community and school based WASH committees, because most of them stay informal.
Training offered to local authorities and communities typically covers some of the following areas: community based planning; construction techniques; appropriate use of WASH facilities and good hygiene practices; maintaining and repairing WASH facilities; supply of spare parts; and organisational and financial management of WASH facilities (composition, role and functioning of committees, tariffs and resource mobilisation as well as record keeping). However, evaluators noticed that training is usually insufficient as they tend not to cover all if these important matters. The DRC 2009, Sudan 2012 and other evaluations regret in particular that participatory planning, community mobilisation, communication skills, and post-construction management were not properly considered. Furthermore, evaluations found that UNICEF does not always use the appropriate methods and tools and enumerate some concrete examples as follows: There is a loss in quality when using a training of trainers approach. Not enough ‘joint training’ are organised with a setting that aims to stimulate a competitive spirit between several committees. Workshop-based training is not appropriate to the targeted audience. More hands-on, continuous training and support is needed. Visuals should replace written long training documents whenever possible.
Quality of programme design and implementation (32 reports from 24 countries)
Many of the sustainability issues examined above could have been avoided or reduced through appropriate programme strategy and management decisions taken by the UNICEF WASH team as well as by its government and implementing partners. Many difficulties and failures identified in these areas are directly attributable to UNICEF. The conception of the intervention as well as implementation process are key in ensuring quality of WASH programming and sustainable results. Cases where there simply exist no experience and knowledge, no commonly recognised standards or good practice, or where the issue lies beyond UNICEF’s control, are rare. The following paragraphs list some of the key programme management related processes and approaches that WASH programmes should consider, based on evaluation reports.
Creating the conditions for sustainability should be an essential component of WASH programmes’s theory of change and logframe. This dimension needs to be paid special attention at the outset of programme design, with specific activities, indicators and targets focussing on the most critical bottlenecks. These should go beyond the naive expectation that, for example, basic policy advocacy and capacity building activities will suffice. In many evaluations, this major weakness is referred to as a “lack of exit strategy” (Ghana, Indonesia and South Sudan 2009, Zimbabwe 2011, Palestine and Sudan 2012, Burundi, Philippines and Bosnia Herzegovina 2013, Pakistan and Mozambique 2014). This term is inadequate, as it seems to suggest that sustainability measures can be considered and implemented at a later stage. Moreover, ‘exit strategy’ is a term that is often used in short-term projects and humanitarian settings, whereas attention to the long-term sustainability should also inform and drive WASH programming in development settings – including scaled up national programmes.
Geographical concentration of all WASH activities should be sought based on a comprehensive situation analysis. It is not always observed in the field because of different government or donor requirements, different past intervention areas or selection criteria, etc. Yet, synergies between water supply, sanitation and school WASH interventions have been proven to enhance sustainability (notably in DRC and Pakistan 2014). Furthermore, it is easier and less costly to conduct regular field supervision and post-implementation monitoring, which are also associated with increased sustainability, when all intervention areas are concentrated. The “LGA (local government area) wide approach” adopted by UNICEF Nigeria is a good practice. Integration with other UNICEF sections and sectors should also be sought because it creates lasting impacts on beneficiaries and the respective interventions can piggyback on each other and reduce operational costs. Again, this is rarely observed in the field for the reasons listed above, and also because of a lack of coordination within UNICEF country offices. The Kenya 2013 report found “vertical sectoral programming enforced by a donor driven programme which paid little attention to defining holistic outcomes for children and women at these key points of convergence” and proposed to set “individual performance criteria within staff performance evaluation report obliging them to engage substantively with health, nutrition and education colleagues in jointly planning for holistic results for children. The same would of course be true for staff within health, nutrition and education units in seeking to regularise cross-sectoral joint planning and review exercises with WASH colleagues”.
Another important step is to make sure the WASH programming is relevant to the needs and priorities of the targeted population. They will likely be ready to engage in field activities and to continue after the end of UNICEF support if it is in their interest and priority. There are different ways of doing so. In some countries, the WASH team designed the whole programme based on a series of situation analyses, field surveys, needs assessments, problem trees and causal analyses, additional specific studies and/or stakeholder consultations (Palestine 2012, Sudan 2013, Ethiopia 2012). Pakistan, Bangladesh and DRC use a participatory local planning approach where the beneficiaries themselves decide on programme activities, outputs and timing with the facilitation of the implementing partner: they apply the PHAST or participatory rural appraisal tools to develop their community action plan (2014 evaluation reports). In other countries, the programmes’ formulation, planning, operation strategy, technology options, activities, implementation process and outputs have been designed and rolled out by the WASH team based on personal experience and ‘good practices’ rather than locally generated evidence, broad consultation, beneficiaries’ participation and in depth reflection. In some of these countries, UNICEF compensates this lack of consultation and analysis by informing beneficiaries about the programme and raising their interest and demand in the initial phase of implementation. As described above (see section 5.5), the DRC ‘Healthy Schools and Villages’ programme and the Mali “WASH in schools” programme are rolled out at a large scale. It would be very challenging to tailor them upfront to the various local contexts. Partly because of this, both programmes have opted for a standardised implementation strategy and process. However, local participation, adaptation and ownership are ensured at a later stage. The target communities or schools are not only informed about the programme, but are also invited to voluntarily apply. The application follows a structured process and is coupled with a formal commitment to actively engage. The expectation is that only those who have a strong need, interest and motivation for the proposed intervention will be enrolled.
Evaluations do not permit for conclusions on the best approach. However, having no approach at all, or an imperfect application of the chosen approach, often lead to weak engagement and ownership on the part of the beneficiaries, and poor sustainability of the intervention. For example, evaluations conducted in 2009 in Indonesia and DRC showed that the lack of ‘participatory culture’ within government agencies and the political economy of beneficiary selection and aid distribution can flaw the outcome of the process.
Furthermore, opting for one approach alone often proved unsatisfactory. The right balance must to be found between various sources of information. Over-reliance on beneficiaries’ perception of their needs and on their planning abilities is risky. Over-reliance on own knowledge and past experience is risky too. General ‘good WASH practices’ may also provide an incomplete picture of what is needed locally. Problem trees and causal analyses do not seem to be often carried out at the stage of programme design – that is to say, prior to the submission of a donor proposal. As a result, some UNICEF WASH programmes have worked on the basis of a theory of change that was not consistent, not comprehensive, or that did not adequately identify hypotheses and anticipate major risks. This led, for instance, to a strong emphasis in several countries on water supply or sanitation intervention without addressing both issues at the same time and, in doing do, ignoring complementarities and externalities22; for example, implementing CLTS without ensuring the availability of an adequate water source in the village for building and maintaining the latrine and for handwashing (evoked in more than half of CLTS evaluations); or improving WASH conditions in schools without encompassing the community, or the other way around; building WASH facilities and the supply side in general at the expense of software activities such as beneficiaries’ mobilisation, awareness raising and hygiene promotion (Palestine 2012, Kenya 2013, DRC 2014); the lack of consideration for rehabilitation needs in Burkina Faso’s schools (2012 report); or the underestimation of training needs to enable sustainable and scaled up results in Indonesia or Kenya (2009 reports).
Beneficiaries must be further involved in the planning and implementation of the programme, including local leaders, women and community health extension workers. This creates ownership and augments the prospects for sustainability. Their participation can and should take various forms.
Local authorities as well as national and subnational ministerial departments should also be part of the decision-making, implementation and coordination process whenever possible. WASH programmes have explored several different programme management models with diverse distribution of responsibilities between stakeholders, encouraged various levels of financial contribution and of participation in field activities, and engaged in a variety of advocacy and upstream initiatives. Some good and bad practices are documented in evaluation reports.
UNICEF WASH teams should make a decision on implementation arrangements and partnerships with government counterparts, CSOs and private service providers based on objective capacity and risk assessments. They were found to be very rarely carried out in the period evaluated, leading to a series of issues that affected programme efficiency, effectiveness and sustainability. Some partners were entrusted with responsibilities that they were unable to take up. Short and incomplete capacity building programmes were offered when continuous, institutionalised training with a broader focus were needed. Inappropriate methods were used. The same issue was observed at local and community level, reducing the ability of relevant player to play their role after the completion of the UNICEF-supported intervention. Insufficient field supervision and control mechanisms were put in place because capacity and willingness of implementing partners were often overestimated, which reduced implementation quality.
In some countries, UNICEF recruits its partners through a competitive process. Interested CSOs submit technical and financial proposals and UNICEF selects the best offer. This process is expected to allow for best quality at lower cost. However, it may actually be an obstacle to long-term collaborations, effective capacity building and therefore to higher quality in programme delivery. The Timor Leste 2015 evaluation illustrates another common issue characterising UNICEF’s relationships with its CSO partners: “In terms of the quality of their relationship with UNICEF as partners, the assessment is less positive. They perceive the relationship to be purely one of sub-contractors in which they are not genuinely involved in joint decision-making. (...) They also feel the need for support in organisational strengthening, such as provided by a number of international NGOs to their local NGO partners. This could include support for developing organisational development strategies and even assistance in diversifying their funding sources, as well as basic training in financial management, accounting and project management”. This is particularly important because the DRC 2014 report shows that “the quality and professionalism of the implementing partner is a key element that affect the ability of the intervention village to maintain its ‘healthy’ status”. Consequently, it must be selected, trained and supervised carefully.
Some evaluations found a disconnect between the software and hardware components of water supply and school WASH interventions, with a long duration between training and IEC activities on the one hand, and construction and provision of supplies on the other. This time gap threatens the adequate utilisation and management of the facilities.
Quality of infrastructure was found to be sometimes poor because of poor design of facilities, lack of detailed standards and technical guidelines, inappropriate planning and supervision, and weak hydrogeological studies – all of which UNICEF is responsible for. Yet, guidance and good practices in these areas are easily available. On the contrary, procurement and contract management are professions in themselves where WASH staff may not always have the appropriate expertise. The supply section in-country can assist at some stages of the process when it is led by the UNICEF office directly but there seem to be a lack of guidance and good practices in this area.
The same is true for tariff setting and pro-poor financial instruments. They are not only key for equity but also for sustainability of WASH facilities and services. However, there are very few guidance and good practices resources that are produced, disseminated and used in the field to address the challenge of capacity to pay and long-term maintenance.
Willingness of beneficiaries to engage and pay is systematically addressed through community mobilisation and awareness raising. The CLTS approach shows that effective approaches exist. UNICEF actively engages in its promotion at all levels. Adaptation to or integration in water supply interventions is not yet systematic however.
Willingness to engage and pay is also driven by internal and external rules, incentives and sanctions. This is an area where UNICEF seems to have insufficiently engaged so far, as reflected in evaluations. Social norms and rules, accountability mechanisms, sector regulation systems, benchmarking, institutionalised competitions and rewards can be powerful tools because they can drive behaviour change and WASH is to a large extent about behaviour change. They can motivate WASH users, service providers, civil servants and field staff in the long run. Willingness is one part of the equation, and capacity is the other. Therefore these measures must be accompanied with continuous post-construction monitoring and follow up support. The Ethiopia 2012 report notes that “what is observed as a gap is the lack of monitoring and evaluation activities once the construction of water points/latrines are completed and handed over to the community/school, or once the community is certified ODF”, whereas the DRC 2014 report demonstrated that the frequency of post-implementation visits and technical assistance is associated with the likelihood of the certified village to maintain its ‘healthy’ status. Both the DRC and Pakistan 2014 reports recommend a six month post-implementation follow up period. Attracting regular and predictable donor funding in addition to the government’s budget will ensure that these sustainability conditions and activities do not cease after the end of the WASH intervention.
Throughout and after implementation, monitoring and evaluation as well as knowledge management and information sharing have a considerable impact on effectiveness, quality and sustainability. As advocated in the Kenya 2013 report, “each staff member should be responsible for analysing possible good practices (or issues needing rectification) and providing the relevant lesson learned or suggestions for improvement to others working in similar areas”. Three reports document good practices in this area. The Zimbabwe 2011 evaluation congratulates the WASH programme for having documented not only “project development and progress indicators, but also lessons learnt, best practices and how key challenges were attended to. All materials developed by the project were extensively shared among the six districts as part of the capacity development and information and experience sharing process. Materials were also shared with WASH cluster members. A ZIMWASH project website (www.iwsd.co.zw) was developed where all important documents were posted. Documenting project process from inception through to planning, implementation, monitoring and evaluation created a rich reservoir of information from which major lessons and best practices were adopted to influence programming”. Mali is the second example where the continuous learning process and feedback loop leads to an annual update of all school WASH programme standards, guidelines, implementation manuals, IEC tools and training material in collaboration with all implementing partners. This process resulted in a constant increase in the quality of software and hardware interventions and in lasting outcomes as evidenced by both the programme and the impact evaluations finalised in 2015. A third example is Pakistan, where “the knowledge management component focused on programme learning and documentation for future design, possible replication and scaling up through several lesson learning workshops, baseline and evaluation studies and action research. Third Party Monitors’ work (contracted for process improvement, verification and feeding into management decision-making) was a major factor in redesigning programme activities based on ground realities that emerged over the course of programme implementation. Regular monthly reviews helped in making the programme more effective whilst ensuring efficiencies were achieved. The baseline findings and learning have been fed into the implementation of Phase 1, whilst the evaluation findings and lessons learnt have been used to inform the design and interventions of Phase 2”, and have been disseminated to other stakeholders. It is worth noting that these three programmes had a dedicated M&E staff member as part of the WASH team.
Contextual and environmental determinants (13 reports from 12 countries)
Contextual and environmental determinants are defined here as factors that considerably affect sustainability of WASH programming and results but on which UNICEF has little or no influence – as opposed to the ones listed above under ‘quality of programme design and implementation’. UNICEF cannot reduce the overall challenge posed by these factors through WASH programming but only anticipate it and, at best, put in place risk mitigation measures. These include: poor governance and corruption at country level; conflicts at community, national and international levels; security issues; political instability; environmental conditions, climate change and natural disasters; and poverty and economic crises. They are very rarely analysed in WASH evaluations. This is a major shortcoming of all reports.
Weak internal controls in government agencies, lack of checks and balances (transparency and accountability), misuse of funds and corruption are known to be obstacles to good management of WASH programmes and services and therefore to sustainability. This governance problem is difficult but known to be widespread in all countries where UNICEF operates. Interestingly, only five reports mention it, of which two in broad terms and for other sectors: two districts in Malawi blacklisted in 2008 by UN agencies for due to failure by government to deploy the directors of finance and internal auditors; and scandal concerning the misuse of DfID funds by the Ministry of Education in Kenya. Two other reports mention corruption in the WASH sector in general as being an issue (Philippines 2013 and Nigeria 2014). In Nigeria, it was stated during interviews that “the community and local government could not sustain the WASH results on their own, even though there are structures at all the levels, due to politics and corruption”. Only the 2010 report on the roll out of CLTS in the WCA region raises the issue more specifically, underlining that “an appropriate and fair ODF certification process is crucial for CLTS up-scaling [and sustainability], but sensitive to misinterpretation and corruption.” Corruption in procurement processes (especially drilling contracts), misuse of UNICEF funds and other forms of mismanagement at institutional and community levels seem to be largely ignored by evaluators and under recognised by UNICEF and its partners. This poses the question whether or not appropriate control and investigation measures are in place.
Conflicts at community level are sometimes briefly evoked as a challenge for the sustainability of the ODF status and of local WASH committees. Conflicts and blockages due to land-tenure issues are evoked in urban and peri-urban WASH.
Conflicts at national or international level, political instability and security issues are mentioned in only four reports: Zimbabwe 2011, Sudan and Palestine 2012, Mali 2015. They reorient governments’ and beneficiaries’ needs and priorities, intensify turnover with government partners and cut communication channels, rise the operational cost of WASH interventions in general and of supplies and logistics in particular, reduce the number of qualified staff and service providers available, limit the possibilities of intensive behaviour change communication and of close monitoring in the field, generate vandalism on infrastructure, engender migrations, and provide a favourable ground for disease outbreaks. They can shatter years of WASH investments. Some countries like Mali that had worked in a development mode for a long time have been suddenly surprised by such a situation. They have adapted the institutional and operation arrangements accordingly, postponed or reoriented field operations to other areas and raised additional funds. However, upstream work and government ownership have been reduced significantly during this period of instability. Other countries are used to working in mixed (emergency and development) settings. UNICEF’s experience and ability to work in both environments is certainly one of its strong assets. Nonetheless, it would be necessary to review evaluations of UNICEF response to WASH emergencies to assess the way that the organisation adequately anticipated and minimised risks. This exercise lies beyond the scope of this meta-analysis.
Overall, the meta-analysis of evaluation reports reveals that little consideration has been given so far to climate change as a threat to WASH sustainability. A response to this threat has been documented in details in only one report (Kenya 2013). In Kenya, the WASH programme has rallied the government, civil society, private sector and other UN agencies to address multi-sectoral environmental and climatic issues through the national WASH coordination platform. A study on climate change related risks and mitigation measures was commissioned, a workshop held to gather successful approaches and building consensus on good practices, and a disaster risk reduction strategy paper developed. As a result, the UNICEF WASH programme refocussed its attention on integrated water resources management, degraded water catchments, water quality, eradication of open defecation and appropriate WASH technologies. The need to further strengthen capacities within UNICEF and in-country on disaster risk reduction and resilience was identified by the evaluator. It is clear that evaluators themselves also need more expertise and guidance on how to take climate change into account in their methodological approach to sustainability. As an illustrative example, the need to increase the depth of boreholes in anticipation of falling water tables is not evoked in any of the reports reviewed.
Economic shocks affect prices and programme expenditures, beneficiaries’ and stakeholders’ needs and priorities, as well as their engagement and ability to contribute. This sustainability risk is evoked in a few reports (notably Zimbabwe 2011) without further development or any assessment of UNICEF’s performance in mitigating it.
Performance of WASH programmes by sustainability determinant can be summarized as follows:
