Report (third draft for comments) Table of content
Yüklə 0,59 Mb.
|
- Bu səhifədəki naviqasiya:
- (third draft for comments) Table of content 1.Introduction 11
- 3.Overview and quantitative analysis of WASH evaluation report 21
- 5.Scalability of WASH programmes: findings 34
- 6.Sustainability of UNICEF-supported WASH programmes: findings 48
- 7.Conclusions and discussion 81
- 9.Good practices and case studies 93
- List of acronyms and abbreviations
- Executive summary Objectives and purpose of the meta-analysis
- Evidence base and methodology
- Definition of the key concepts
- Conclusions on WASH programming
- Conclusions on WASH evaluations
- Improving the quality of WASH programming: Recommendations
- Improving the quality of WASH evaluations: Recommendations
 Equity, scalability and sustainability in UNICEF WASH programming: A thematic meta-analysis of UNICEF’s WASH evaluations 2007-2015 Report (third draft for comments) Table of content 1.Introduction 11 1.1.Background 11 1.2.Purpose 11 1.3.Objectives and intended audience 11 1.4.Structure of the report 12
2.1.Methodological approach 13 2.2.Defining key concepts 13 2.3.Selecting the evidence base 16 2.4.Analysing the content of the evidence base 17 2.5.Contributors and timeline 18 2.6.Limitations 19
3.1.Coverage of evaluation reports 21 3.2.Quality of evaluation reports 21
4.1.Treatment of equity issues in evaluation reports 23 4.2.Overall programmes’ performance 23 4.3.Findings by stage in the programme cycle 23 4.4.Findings by population group 25 4.5.Findings by type of WASH intervention 27 5.Scalability of WASH programmes: findings 34 5.1.Treatment of scalability issues in evaluation reports 34 5.2.Overall programmes’ performance 34 5.3.Findings on ways of programme scaling up 34 5.4.Findings by determinants for programme scalability 36 5.5.Findings by type of WASH intervention 39 6.Sustainability of UNICEF-supported WASH programmes: findings 48 6.1.Treatment of sustainability issues in evaluation and sustainability check reports 48 6.2.Overall programmes’ performance 48 6.3.General findings by sustainability determinants 49 6.4.Findings by type of WASH intervention 64
7.1.On WASH programmes 81 7.2.On WASH evaluations 84
8.1.Recommendations for UNICEF’s future global WASH strategy 87 8.2.Recommendations to the WASH section at HQ and regional offices levels 88 8.3.Recommendations to the WASH sections at country level 89 8.4.Recommendations to the Evaluation Office and the Regional Evaluation Advisors 90 8.5.Recommendations to the evaluation commissioners at HQ, regional and country office level 91 9.Good practices and case studies 93 9.1.On equity 94 9.2.On scalability 95 9.3.On sustainability 96 10.Annexes 97 10.1.List of evaluation and sustainability check reports 97 10.2.Detailed analysis of the evidence base 97 10.3.Matrices of content analysis 97 List of acronyms and abbreviations C4D Communication for development CATS Community approaches to total sanitation’ CEE/CIS Central and Eastern Europe and the Commonwealth of Independent States CLTS Community-led total sanitation CSO Civil society organisation DANIDA Danish International Development Agency DFID UK Department for International Development DRC Democratic Republic of Congo EAP East Asia and the Pacific ESA Eastern and Southern Africa FGD Focus group discussion GEROS UNICEF’s Global evaluation reports oversight system GTZ German Technical Cooperation Agency (currently named ‘GIZ’) IEC Information, education, communication LAC Latin America and the Caribbean Logframe Logical framework M&E Monitoring and Evaluation MENA Middle East and North Africa NGO Non-governmental organisation O&M Operation and maintenance ODF Open defecation free OECD / DAC Organisation for Economic Co-operation and Development / Development Assistance Committee PHAST Participatory Hygiene and Sanitation Transformation PPP Manual Programme Policy and Procedure Manual RO Regional office SA South Asia SDG Sustainable development goals SIDA Swedish International Development Agency SWAp Sector-wide approach UN United Nations UNEG United Nations Evaluation Group UNICEF United Nations’ Children Fund USAID United State Agency for International Development USD United State dollar VSLA Village Savings and Loan Association WASH Water, sanitation and hygiene WCA Western and Central Africa WHO World Health Organisation
The main research question for this thematic meta-analysis of UNICEF’s Water, Sanitation and Hygiene (WASH) evaluations is: What is the performance of UNICEF-supported WASH programmes in terms of equity, scalability and sustainability, what may have affected performance, and what are the lessons learnt? The primary purpose is to provide a critical review of and to learn from the current global WASH Strategy (2006-2015) in order to inform the development of the new WASH strategy that will guide UNICEF’s action toward the Sustainable Development Goals (SDGs) during the period 2016-2030. It is also expected that the meta-analysis will contribute to the improvement of the quality of WASH programming in the field. Its secondary purpose is to feed into some of the current, most vivid discussions around aid, both internally within UNICEF and in the broader WASH sector, by bringing new evidence-based insights. The primary audience is therefore the WASH section at UNICEF HQ and at regional and country office levels. The secondary audience are the broader WASH sector globally, WASH evaluators and evaluation managers both within and outside UNICEF, as well as non-WASH development experts.
UNICEF-supported WASH programmes encompass all types of water supply, sanitation and hygiene promotion interventions that are listed in the current WASH strategy and that UNICEF country offices are typically engaged in. In practice, most reports reviewed examine rural water supply interventions, community-based total sanitation (CLTS) and hygiene promotion interventions in rural areas, and/or school-based WASH programmes. Included in the meta-analysis are evaluation reports of UNICEF-supported WASH programmes commissioned by UNICEF between January 2007 and July 2015, as well as sustainability check reports.1 Excluded are mainly evaluations whose quality has been rated unsatisfactory and evaluations of WASH responses to punctual emergencies that are not intended to be scaled up and sustainable. Based on these criteria, out of the 112 reports initially identified, 74 were retained for the analysis. They cover all UNICEF regions, all countries with major WASH programmes, and all types of WASH intervention. The analysis was done in a way that allowed reviewers to make comparisons, detect trends and patterns emerging from the overall evidence base, identify singularities, and find informative cases related to each thematic focus for each type of WASH interventions. This process reduced the risk of using anecdotal, inconsistent or potentially weak evaluation evidence to draw general conclusions. Therefore, the strength of the evidence base is considered good overall. The analysis and report drafting were carried out by the UNICEF Evaluation Office, with support from a small team of consultants in the analysis phase.
In the context of this meta-analysis, equity means that the resources, goods, services and opportunities produced by the WASH programme shall primarily benefit the deprived, vulnerable or marginalised groups, based on their needs and priorities, with the intention of reducing existing inequalities between them and more favoured populations. This definition implies that an equity lens is used at each stage of the WASH programme cycle. The population groups of greatest interest are the following: the poor and most deprived; women and girls; the hard to reach; elderly and disabled people; population at risk of climate change; other categories of marginalised or vulnerable population (remote and marginalised groups, religious and ethnic minorities, slum dwellers, pastoralists, aged and disabled people, and people affected by HIV). The meta-analysis explores the extent to which the needs and priorities of these population groups were taken into account at each stage of the programme cycle. Scalability refers to the ability and likelihood of a WASH programme to expand from a limited scale to a larger reach while retaining effectiveness and efficiency. The upscaling process can be the result of a spontaneous diffusion in the field, of a replication strategy developed by UNICEF, its implementing partners, the government, or other development agencies, or of an institutionalised mainstreaming. Several factors influence the scalability of a WASH intervention. Both scalability and actual evidence of upscaling are analysed based on evaluation reports, for WASH programmes in general, and for rural water supply, CLTS, hygiene promotion and school WASH interventions in particular. Sustainability is the ability of prevailing national and local structures, processes and people to continue their role and functions after the withdrawal of all forms of assistance from the external development agency. As a result, the WASH programme outputs, outcomes and impacts would persist. Depending on when the evaluation was carried out along the programme timeframe (during programme implementation, at completion of the programme, or after the end of the programme), this meta-analysis looked either for the documented presence of the conditions known to be associated with WASH sustainability (likelihood / prospects for sustainability), or for evidence of WASH activities and results having been maintained during a significant period of time after donor funding has ended (actual sustainability). A commonly agreed definition of what a ‘significant period of time’ means for a WASH intervention does not exist, but was set to at least one year for the purpose of this meta-analysis based on data availability. Conclusions on WASH programming Equity The meta-analysis found that the performance of UNICEF WASH programming in terms of equity is mixed. WASH interventions have strong equity benefits. In terms of population groups, the effect of UNICEF-supported WASH programmes on the lives of women and of the poorest populations is relatively well documented in the evaluation reports and consistently positive especially for women and girls, mainly based on qualitative evidence. The performance for the other categories of vulnerable population is more limited or unknown. However, the integration of equity principles in UNICEF programme designs and implementation strategies in the field could be further strengthened. When a positive equity effect is reported, it is to a large extent an ‘automatic’ effect. It is due to the fact that deprived and vulnerable populations are typically the ones who suffer the most from the lack of improved access to WASH, and beneficiary communities tend to apply internal solidarity mechanisms in favour of those who need it most. But UNICEF’s added-value in supporting the most vulnerable people is too modest, especially where internal equity-related awareness mechanisms do not exist. Based on the evidence collected, UNICEF’s contribution to supporting equity through WASH programming seems to be often restricted to the promotion of women’s participation in local management structures (water point management committees, village sanitation committees, and school health clubs) and sex-separated latrines in schools, health centres and public places. There is still room for improvements in terms of specific support provided to the other population categories. Main shortcomings identified are the following. Only a quarter of evaluations noted that the geographical targeting of vulnerable and marginalised groups was adequate. This is an issue especially for water supply and WASH in schools with other criteria than equity being taken into account when choosing areas of interventions. In-depth equity analysis aiming to inform the design and implementation of UNICEF WASH programmes are mentioned in only two reports. Involvement of the various categories of beneficiaries to ensure at an early stage that their needs and priorities are integrated in programme design is also uncommonly found in evaluations; evaluation evidence suggests it happens at a later stage, during siting, tariff setting and decisions around management arrangements. Rural water supply and sanitation programmes make use of a limited number of pro-poor financial and non-financial instruments. Equity-lensed monitoring and evaluation systems are not in place. These shortcomings make it challenging to measure whether UNICEF WASH programmes are actually contributing to the reduction of inequities. But, evidence from 12 evaluations in Sub Saharan Africa, Asia and Latin America shows that while WASH programmes improve the lives of women, the poorest and the most vulnerable, they don’t alone cannot fundamentally empower them socially because they fall short in sustainably influencing dynamics of inequities in the local decision-making process, including on WASH issues. In addition, the evidence (including UNICEF evaluations that refer to the broader academic literature) for the impact of WASH in schools on girls’ education is not yet robust and consistent; therefore, no conclusions are possible. Globally, CLTS interventions have been found the most equity-sensitive of all types of WASH interventions. They typically focus on small rural communities with higher rates of open defecation and poverty. They encourage a large participation from all villagers through the community-based approach and adopt facilitation methods that are well adapted to the less educated among them. They promote the use of local construction materials and techniques that require no cash disbursement. They benefit everyone (the cessation of open defecation must be universal in the village), especially women as well as the aged and disabled people. Overall there is still room for improvement for UNICEF to integrate the equity agenda consistently and adequately in its WASH programming, especially in water supply and WASH in schools. More reflection is needed to provide WASH ‘leaving no one behind’.
The performance of UNICEF WASH programming in terms of scaling up is moderate, except for CLTS where performance is higher. Spontaneous diffusion of WASH interventions and results is not well documented. Evidence shows however that such a diffusion is limited, mainly because on the one hand WASH interventions are usually technology and cost intensive, and on the other hand improvements in sanitation and hygiene related behaviours and capacities require intensive and repeated efforts that are not easily internalised by beneficiaries and stakeholders without external support. While encouraged or organised replication can strongly contribute to the efficiency and impact of WASH programmes, it is neither formally included from the outset as an objective in the design of most programmes, nor integrated in programme logical frameworks (logframes) and intervention strategies described in evaluation reports. It could be better planned for and driven not only by UNICEF but also by governments and donors. Evaluations in six countries found that UNICEF played an important role in bringing together stakeholders who previously worked in silos in development settings, to share experiences and tools developed in regular programming and to encourage their replication. Cases like these illustrate the strength of UNICEF being both an actor in development and emergency settings and taking on the role of WASH cluster coordinator in case of an emergency. The meta-analysis suggests that the main constraint for water supply and WASH in schools and health centres interventions is not only the unavailability of financial resources but also the lack of effective and scalable approaches and the lack of absorption capacity in-country. More funds can be made available by governments or external support agencies when both of these preconditions are met, as demonstrated by CLTS. The CLTS approach, which is being scaled up successfully by UNICEF, has not only benefitted from low implementation costs. It also benefitted from both an operational strategy that is easy to communicate and does not require a high absorption capacity, as well as intense advocacy and mainstreaming efforts on the part of UNICEF. The scale of UNICEF’s water supply interventions and, to a lesser extent, of WASH interventions in schools, is still significant. However, the likelihood of these types of intervention to be taken to a larger scale will probably depends on UNICEF’s ability to draw the lessons from the CLTS experience and develop innovative approaches. In terms of advocacy and mainstreaming as strategy for scaling up, one area of reflection that seems to have been neglected so far is incentive mechanisms. Evaluations have identified only a limited number of cases where UNICEF has helped governments put in place organisational reforms, sound administrative procedures and routine activities aiming to create WASH-related obligations, pressure or motivation factors for all stakeholders at national and local levels. There is little evaluation evidence on successful urban WASH programming. Evaluations reviewed outline the challenges of implementing infrastructure-based sanitation interventions at scale in urban and peri-urban settings because of higher unit costs and land tenure issues. The effectiveness and scalability of CLTS in these areas is also questioned notably because social capital, buy-in and ownership tend to be weaker than in rural communities. No example of success has been reported in the area of (peri-)urban water supply delivery, management models, or tariff setting. In conclusion, greater achievements are observed in taking CLTS to scale than for other types of WASH interventions. A number of options need to be explored by UNICEF WASH in order scale up results to significantly contribute to the achievement of universal access to water and to WASH in schools.
The performance of UNICEF WASH programming in terms of sustainability, overall, is moderate, except for the promotion of hygiene behaviours in general, and of handwashing and drinking water treatment in particular, where it is weaker. The sustainability of drinking water provision in communities and schools, most often measured by the functionality rate of the water point at any given time (without considering the age of the infrastructure), ranges between 60% and 90% depending on the type of technology, the geographical area, the season etc. It slowly decreases over time. The longer term sustainability of water quality and the level of capital replacement is not well documented. The main factors explaining cases where the water point is found dysfunctional are the following: the planning phase did not take into account the population size, leading to overuse of water points; the field supervision during the construction phase was insufficient, causing low quality infrastructure and technical defects; and the community management (including cost-recovery mechanisms) and the monitoring and regulation system in place were weak, explaining the lack of preventive maintenance and sometimes delays in repair. The level of sustainability of latrine use one to five years after open defecation free (ODF) certification is comparable to the one of water supply, with also a progressive decrease over time. The concern widespread in the literature around the sustainability of CLTS does not seem substantiated by the quantitative evidence available, unless the practice of handwashing is taken into account in this measure as a criteria for ODF status. Based on the evidence reviewed, actual practice of handwashing is the CLTS component that is driving its level of sustainability down. Moreover, the meta-analysis suggests that more attention is paid to technical and financial determinants as an explanation for reversion to open defecation than to institutional determinants and the enabling environment while these are equally critical. Additional critical barriers include the lack of willingness of households to upgrade their latrines, and the time and efforts needed to support lasting changes in behaviour through frequent follow up visits by implementing partners, technical back-up support and incentives. Hence, mitigation strategies consisting of providing financial support to the poorest households for improving the quality of their latrines and strengthening the supply side might not be an appropriate or sufficient answer to the challenge of ODF sustainability. The hygiene challenge, the lack of attention given to the institutional environment – administrative monitoring and incentives, continuous training and supervision, etc. – and the short timescale of implementation are equally found in school WASH. The effective and sustained adoption of handwashing is the greatest challenge for UNICEF. According to the 20 reports that quantified handwashing it was reported that between 5% and 25% of people wash their hands with soap or ash even shortly after intensive promotion. It is a concern given its demonstrated low cost and high health impact. It is also a cross-cutting issue for all WASH interventions in communities, schools as well as health centres. While there are many determinants of sustainability in WASH, the two most problematic ones identified in this meta-analysis are first the institutional factors: lack of governments’ leadership and capacity, weak enabling environment, deficient regulation and incentive mechanisms, lack of national budget allocated to routine post-intervention monitoring, and turnover of staff. They are difficult to influence and would take time to address. One area of reflection and potential improvement is the integration of WASH within existing administrative structures and procedures, and the institutionalisation of incentive mechanisms. The second major challenge is the quality of programme design and implementation. This includes limited timeframes, and weak anchoring in government agenda and management bodies, and lack of geographical concentration of WASH interventions and of integration with other sectors. This can be addressed more easily, and would have immediate consequences for the sustainability of UNICEF WASH interventions. In conclusion, as sustainability is now given priority both in the WASH sector and globally, concrete actions could be taken to place it more at the core of UNICEF global WASH strategy and field programming, not only in water supply and sanitation but also and foremost in hygiene promotion. Overall conclusion The highest performing WASH intervention in terms of equity, scalability and sustainability is CLTS. UNICEF interventions in water supply and WASH in schools and health centres are less performant especially in terms of scaling up. Regarding these two types of interventions, UNICEF needs to continue its reflection and learn from its successful experience with CLTS. The key dilemma to be resolved is: Given the new global ambition set in the SDGs, the limited absorption of national governments and the fragile environment in which it operates, how can UNICEF achieve results at scale without compromising on quality of programming and sustainability? Conclusions on WASH evaluations General conclusions Weak or incomplete evidence is generated by UNICEF through WASH programme evaluations and sustainability checks on equity, scalability and sustainability. Evaluations should be treated as a critical means of assessing the organisation’s performance, identifying relevant approaches and good and bad practices, and improving the quality of programme design and implementation in the field as well as the appropriateness of national and global WASH policies and strategies. However, this can only happen if the quality of evaluations increases and if findings feed back into management decisions and actions. A major challenge is the methodological approach used in WASH evaluations and sustainability checks, both weak and inconsistent. Indicators, sampling strategies and data collection methods and analysis often lack robustness. Obvious bias in the data collection methods, notably over reliance on self-reports and informant interviews, and lack of triangulation are rarely acknowledged in the limitations section of the reports – when there is one in the first place. Therefore, accuracy and reliability of findings are not always ensured. A second challenge is the weakness of monitoring and evaluation (M&E) frameworks in WASH programmes and the absence of baseline data and of a counterfactual allowing for a comparison between before/after and with/without the intervention. As a result, evaluations are not always able to directly attribute outcome and impact-level results, especially related to behaviour change and broader health and socio-economic benefits, to the UNICEF-supported WASH programme. In other words, they are often unable to state what difference the programme really made. A third challenge is the effective implementation of corporate procedures related to the quality assurance and use of evaluation findings. In some cases, evaluation reports had been misclassified as reviews, studies or surveys – or the other way around. Appropriate quality assurance processes were not systematically applied. Even when reports had been correctly classified, they were not adequately reviewed by both WASH and evaluation experts. More quality assurance would have probably helped improve the quality and usefulness of a number of evaluation reports. Furthermore, in terms of utilisation, reading evaluations that have been conducted in the same country with a few years’ interval reveals that appropriate corrective action was not taken as a result of the first evaluation. This raises questions around the monitoring of the management response to evaluations and of their effective implementation. More specific issues related to the treatment of equity, scalability and sustainability in evaluation (and sustainability check) reports are summarised in the below paragraphs. Equity Equity is addressed in most evaluation reports. This trend has increased since 2010. However, a definition of equity is not provided in any of the reports, and the concept is almost never unpacked. For example, evaluations do not differentiate between equity (or gender) sensitive versus equity (or gender) transformative approaches. They do not define how equity should look like in WASH programming. Moreover, evaluations gather insufficient data on the demographic, economic and social context, e.g. pockets of poverty or existence of marginalised ethnic groups, and limited information on the specific strategies aiming at reaching these population groups. These shortcomings lead to equity being incompletely and imperfectly assessed. The situation of population categories other than women and the poor is generally overlooked. The performance of WASH programmes in terms of equity is not systematically examined by following the successive phases of the programme cycle. The strength of the analysis varies depending on the programmatic phase that is looked into. In general, the level of analysis is the weakest when it comes to assess the initial situation analysis undertaken by UNICEF, the M&E system in place, and the impact effect of WASH interventions on the actual reduction of inequities. Scalability Performance in programme upscaling and scalability is documented in slightly more than three quarters of evaluation reports. But this analysis is not developed in a dedicated section of the reports because it is not a standard, stand-alone evaluation criterion recommended by the Organisation for Economic Co-operation and Development (OECD) for evaluating development assistance. As a result, information is often scattered between the sustainability, efficiency and relevance sections, and more rarely in the effectiveness section. Furthermore, there is a confusion between these different concepts, especially between scalability and sustainability, because evaluations do not define the notion of scalability as a basis and starting point for the analysis. Information is overall scarce. Reports generally only document cases where the programme has been replicated by other development partners or mainstreamed in national policies. Evidence of spontaneous diffusion are seldom investigated. Evaluations reviewed do not highlight as a negative finding cases in which WASH programmes have had a very limited scale and did not try to expand, as though scaling up was not an important objective of WASH interventions in development settings. Finally, the presence of the various factors that are drive or constrain programme upscaling is almost never examined in a systematic approach. Sustainability Sustainability is a standard OECD’s Development Assistance Committee criteria for development evaluations and is used as such in all UNICEF WASH evaluations. It is typically addressed in a dedicated section. The roll out of sustainability checks in Sub Saharan Africa as promoted by the Government of the Netherlands is also a positive sign indicating that the sustainability discussion will be further informed by evidence. However, the same issues arise as for equity and scalability: the notion of sustainability in WASH is not defined and unpacked in any of the reports, even in the sustainability checks. For example, what sustained results would mean for WASH programming, and what the difference is between actual sustainability and likelihood for future sustainability is not made clear. After how many years can we really start to say the intervention was sustainable? What rate of water system breakdown or ODF slippage is tolerable after 1 year? After 5 years? When analysing the prospect for sustainability, what sustainability determinants should be examined? Therefore, the analysis of sustainability considers only a few number of sustainability indicators. It rarely looks thoroughly at all the components of the evaluated programme. This shortcoming is likely not specific to UNICEF WASH evaluations, however. Sustainability checks, which are specific to UNICEF, do not use common indicators and benchmarks, a well-defined sampling strategy and rigorous data collection methods. They lack baseline data. Consequently, data are often neither reliable nor comparable. They are difficult to use for global and comparative analysis. Evaluations and sustainability checks mostly focus on the functionality of WASH facilities. The current evaluative approach to sustainability is insufficient to capture performance at service and (sub)national levels. It also fails to examine the status of the main determinants known to be associated with sustainability, especially the environmental / contextual factors and the financial factors.
The recommendations are summarised below, and further developed in the main body of the report. Recommendations for UNICEF’s future global WASH strategy
To the WASH section at HQ and regional offices levels
To the WASH section at country levels
Improving the quality of WASH evaluations: Recommendations To the Evaluation Office and Regional Evaluation Advisors
To evaluation commissioners at HQ, regional and country office level
Yüklə 0,59 Mb. Dostları ilə paylaş:
|