2. Key Factors Affecting Implementation and Outcomes
2.1 Project Preparation, Design and Quality at Entry
Project Design. The project design was based on the lessons of APL 1 outlined in its ICR (ICR000073). In particular, it was emphasized that: (1) the lack of a clear implementation strategy that would ensure results, among other factors, caused difficulties in the first phase of the program, which were overcome by using evidence-based research to focus on results; (2) political volatility was detrimental on project implementation and impact and, while ministerial influence was necessary; it was also important to implement project activities in coordination with the areas of MINSA responsible for specific project components; (3) there was a need to build institutional and managerial capacity at regional level before transfer funds directly to these local management units; and (4) intercultural strategies were vital when beneficiaries belonged to diverse indigenous groups and community participation was key to ensure sustainability. In addition, the second phase of the program, including its activities and the Results Framework (RF), was informed by a comprehensive feasibility study finalized in 2008 (with a baseline for the indicators taken in 2005).
Due to a strong interest of the GOP at the time of appraisal on infrastructure investment, the project was largely focus on upgrading health facilities in the nine regions, where lack of infrastructure was identified as an important barrier to service access. Yet, these infrastructure investments were used as entry point to promote broader sectoral reforms and the use of evidence-based policy. These “soft interventions”, which were at the core of PARSALUD I, were less prominent in PARSALUD II; however, they are fundamental in ensuring sustainability of progress.
Project Preparation. Preparation of APL 2 started promptly, even before the closing of APL 1. As was the case for the first phase, APL 2 was also co-financed by the World Bank and the IADB, and therefore, project preparation was conducted in close collaboration with the IADB team3. However, a three years gap stands between the end of phase I and the effectiveness of the project supporting phase II. Changes in political priorities and leadership due to frequent changes in Government were the main causes of this delay. When a new Minister of Health was appointed in October 2008, an opportunity window opened up and project preparation regained traction. The project was negotiated in December 2008 and approved by the Board of Directors on February 17, 2009. The signing of the Loan Agreement only happened 9 months later, on November 16, 2009 and the project was declared effective on December 15, 2009.
Quality Enhancement Review (QER) and Decision Review Meeting (DM). The project underwent a QER in May 2006; the DM was held in December 2008. During both meetings, the element of the project that was mostly appreciated by the reviewers was the cultural adaptation of all the planned activities. Some of the issues and recommendations raised during these meetings not only revealed to be crucial to determine quality at entry, but were eventually found to be critical during implementation. These included:
Establishing clearer links with the previous APL and, more broadly, better explaining how the new operation would fit within the Peruvian institutional environment, which had changed from the end of the first phase of the program and, even more, from its original conceptualization;
Strengthening the M&E system by: (i) reducing the number of indicators, but establishing a clear results chain from activities to outputs and outcomes; (ii) reducing the number of data sources from which the indicators would be derived; and (iii) ensuring that the counterpart had the capacity to monitor the RF;
Assessing more realistically the risks deriving from the institutional environment which are outside the scope of the project, especially with regard to the links with SIS and the dependence upon approvals from the National Investment System (Sistema Nacional de Inversión Pública, SNIP). During implementation, it became clear that SIS had gain much more independence than anticipated and coordination with PARSALUD gradually reduced. SNIP, which was created in Peru in the early 2000s, was rather rigid in its conceptualization of investment projects, with a non-participatory decision-making process and long approval time of programs, thereby compromising the possibility to efficiently make any changes to the projects during implementation. Furthermore, its clear preference for financing infrastructure investment left very little margin to incorporate other investment approaches, such as those adopted by APL 1, despite their proven effectiveness.
2.2 Implementation
Implementation of the project did not suffer from any major complications. The project had a slow start. While activities related to Components 1 and 3 started immediately, delays in disbursement and execution related mainly with the infrastructure investments under Component 2. Once construction works began and medical equipment started being purchased in 2011, project disbursement picked up towards the end of 2012.
As a result of the nature of the activities, project implementation was characterized by a high volume of transactions. Given the commitment to co-finance all civil works of the PARSALUD program with 6% funding coming from loan resources (i.e. 3% from the Bank and 3% from the IADB) and the remaining 94% from domestic resources, all procurement processes followed the Bank’s procedures. This was highly desirable from the perspective of MINSA, given the stricter Bank procurement guidelines, and it reflected a general trend in Peru at that time ― when the GOP was seeking external resources to fund rather small portions of broader national investment programs to benefit from streamlined procurement processes and technical assistance. Yet, this posed stress on the task team for the supervision of all transactions related to the program, efforts that were, therefore, disproportionate with respect to the resources committed with the loan.
While the program was successful in achieving its broader goals of reducing maternal and infant mortality and chronic malnutrition in children under 5, progress on the indicators in the RF was mixed during the life of the project. Some of the targets were achieved even before the MTR in February 2013, while others had a more fluctuating trajectory. To some extent, this was related to the delays in construction works, which shortened the time horizon available to see the impact of the infrastructure investments on the selected health outcomes and outputs. In other cases, the trend of some indicators reflected those at the national level, such as in the case of the prevalence of anemia among pregnant women. The delays in progress on some indicators motivated the downgrade of the Overall Project Implementation (IP) rating from satisfactory to moderately satisfactory in 2013, rating which was then kept in consideration of the disbursement delays ― eventually, the project disbursed 80% of the planned amount.
The Mid-Term Review (MTR) in February 2013 identified some of the challenges and correcting measures, including the needs to better coordinate with other relevant units within the MINSA and regional governments, strengthen M&E, increase loan disbursement (by then only 35%), modify the RF to reflect the actual starting and end date of the project, and support the reform process within MINSA by increasing capacity of and coordination with the regional and levels. Some of these recommendations were immediately taken on board, including for example strengthening the M&E function within PARSALUD. Others were only partially addressed during the life of the project.
Overall, the following implementation strengths were identified:
Project Implementation Unit (PIU)’s capacity and commitment. The PARSALUD PIU was reconstituted in 2009, after a period of two years from its closure after the program’s first phase. The capacity of the PIU for project coordination was assessed as satisfactory and the unit was generally appropriately staffed (see Annex 12). The role of project coordinator was stable (two main coordinators and two acting for a very brief period of time). The project coordinators and many of the key personnel were very committed and remained within the team throughout project implementation, ensuring continuity and supporting improved capacity of the PIU. The PIU was found to be very proactive in a number of areas. For example, the PIU accepted all communications and documents to be sent electronically to the IADB and the Bank, which is not common practice in Peru. The PIU developed and made available checklists to constructors to ensure environmental safeguards were abided by and monitored compliance independently. Finally, the PIU worked closely with each Direccion Regional de Salud (DIRESA) to support local and regional-level interventions (such as the EPPES) and to strengthen capacity at the regional and local level based on the needs of each region.
Cultural adaptation of interventions. The systematic strengthening of health rights and empowerment of the population in rural areas about social participation in health through the EPPES and DIA campaigns were very well received by the local communities. These initiatives, together with the prior consultations held with local communities before the start of all civil works under PARSALUD, helped regional administrations build capacity for intercultural strategies, including communication in local languages (e.g. Quechua and Aymara). As a result of those culturally-sensitive interventions, coverage from SIS increased in the lowest quintiles in the Project areas.
Contribution to the Identity and Insurance Rights movement. PARSALUD was very active in catalyzing efforts to support the Derecho a la Identidad y Aseguramiento (DIA) for the health sector. This was an intersectoral initiative, in collaboration with the RENIEC and civil society, and with a strong regional and local commitment, to which PARSALUD contributed by supporting a campaign to promote the issuance of the Live Birth Certificate to children under 3 years of age and the National Identity Document to pregnant women and mothers.
The following implementation weaknesses were identified:
Political changes within MINSA and regional governments. Four Ministers of Health changed during project implementation. Despite posing some concerns about stability of political commitment for the project and its key staff, eventually political changes at the national level did not substantially impact on project implementation. On the other hand, frequent political changes in the regional governments meant that PARSALUD PIU had to constantly re-engage with new administrators and staff, requiring duplication of efforts for capacity building at the local level.
Delays in civil works and consultancies. As of December 2012, after 2 years of implementation, about 27% of project's civil works were delivered, with heterogeneous patterns among the regions. Logistic difficulties for timely civil works completion were associated with difficult climate and access to sites, as well as with a few cases of collusion, properly addressed by the counterpart. Delays in hiring consultants were due to, among others, frequent changes in regional administration, scarcity of professionals adequately qualified for some tasks, delays in approval processes from the regions, and political attention diverted to sudden public health emergencies (e.g. pneumonia in Puno and dengue in Ucayali). Despite all delays, almost all planned civil works were completed before the project closing date, with a few being delivered in 2016.
Underestimation of the impact of other programs or initiatives on progress on the PARSALUD indicators. In some cases, project indicators showed irregular progress. This was partially due to other government programs providing incentives contrasting with those provided by PARSALUD (e.g. distribution of formula milk which at times affected exclusive breastfeeding of infants), and partially with changes in procedures for the registration with SIS (e.g. requirement of national identification document to register with SIS, which negatively affected the number of affiliates).
2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization
Design. The project’s Results Framework (RF) was derived from the PARSALUD program RF, which was based on a feasibility study (FS) finalized in 20084 and approved as the program proposal by the SNIP.
The indicators in the RF reflected all parts of the PDO. The PDO-level indicators (Key Performance Indicators, KPIs) focused on improving practices at the household level (PDO 1) and on strengthening the health service network (PDO 2). The Intermediate Outcome Indicators (IOIs) focused on PDO 2 and 3 (supporting MINSA’s government functions). The first three components of the project were also aligned with the three parts of the PDO. Given the funds allocation, KPIs and IOIs focused largely on PDO 2 and Component 2 (strengthening of health services networks – supply side). Component 3 (governance and financing) is reflected by a small number of IOIs, most of which were dropped in 2014.
Baseline data as of 2005 was available for all but three indicators, based on the FS. Targets were set against that baseline by imposing improvements greater than the expected improvements based on historical trends. However, delays in project preparation and effectiveness made the baseline and the targets outdated, but neither of these were revised. If unable to change the baseline and/or the targets at the time of appraisal, due to time constraints and lengthy government processes, within the context of the first restructuring in 2011, when data from 2009 was becoming available, these should have been used to update the baseline and the targets.
Implementation. The project indicators were monitored using government’s systems and surveys. This had the advantage of not creating a parallel system. However, the PIU had to request or download data from different institutions responsible for data collection before being able to analyze it. There was a time lag of one year between the data collection and the data availability, so that data for a given year became available only in the following calendar year.
Due to unavailability of data at the time of the ISRs, two KPIs only started being monitored in December 2014 (ISR 11, with data of December 2013). Given that they had not been monitored, five IOIs were dropped during the restructuring in 2014. However, data was available for two of those, which were reintroduced for the purpose of the ICR. The other three indicators had not been clearly defined and monitoring was therefore problematic. Unfortunately, all three of those IOIs measured progress against the same part of the PDO (3). Since the PARSALUD PIU monitored more indicators than those monitored by the Bank, additional intermediate indicators might have introduced to replace the problematic IOIs to better assess improvements on PDO 3 ― for example during the first restructuring in 2011 or immediately after the MTR in 2013.
Within the PIU the M&E Unit originally included two specialists, one focused on Monitoring and the other on Evaluation. Following suggestions from the MTR, the M&E function was strengthened and the original unit was split into two to focus and strengthen each area ― supervision of program performance, and management of scientific evidence for enhanced effectiveness of the overall program (see Annex 12).
Utilization. Once data was processed, the M&E team analyzed data for each indicator and informed the technical team of the trends. If needed, the local coordinators within the DIRESA were contacted to understand the reasons for the variation in the indicators, especially with regard to birth, death and maternal anemia. However, given the difficulties in monitoring the full results chain and the delay in obtaining information on the indicators, data was not used by the local level to inform decision-making or revise practices and procedures in real time. It was used by the PIU to promote studies that supported increased knowledge and evidence-base policy, informing the design of technical guidelines and regulations that improved MINSA’s regulatory capacity.
2.4 Safeguard and Fiduciary Compliance
Safeguards. Given that the project (environmental category B) triggered the Environmental Assessment (OP/BP/GP 4.01) and the Indigenous People Safeguard Policies (OP 4.10), environmental and social safeguards were monitored. The Environmental Assessment was conducted in 2005; the Indigenous People Plan (IPP) was prepared in 2006. In 2013, specific missions assessed the compliance with environmental and social safeguards and proposed corrective recommendations where needed. More targeted safeguards supervision missions were conducted in 2013 to review implementation of the action plans in line with the EA and the IPP. The implementation of the activities under an intercultural approach related to Component 1 (including the EPPES, the DIA, and the prior consultations to IP for infrastructure construction) were rated as satisfactory. Similarly, the implementation of environmental safeguards was rated satisfactory, given the proactivity of the PIU in promoting the use of checklists for solid waste and water management during infrastructure construction under Component 2.
Financial Management. The counterpart’s financial management performance was considered generally satisfactory. The PIU was appropriately staffed from the start and, despite delays in systematically adopting the official system for managing transactions used in all implementation units in the public sector (Sistema Integrado de Administración Financiera, SIAF), financial reports were timely and were found to be of satisfactory quality. Audit reports were provided on time and there were no qualified opinions. Until 2013, disbursement was very slow (39% in August 2013). Problems were related to a number of factors, including previous delays in civil works as a consequence of inadequate planning and due to the remote project locations, which did not offer incentives for enterprises to participate in the bidding process; and the contractors’ noncompliance with contractual clauses. The GOP established a condition of Pari-Passu for all civil works (6% of external resources), which limited the scope for accelerating disbursement of the loan. Finally, deferred payments for civil works at the end of the project (Ocongate Health Center and Health Center Chuquibambilla) and the cancelation of the scheduled execution of large amounts of consulting services (i.e. Diplomado APS PROFAM and Sistema Nacional de Sangre Segura) expected to be contracted before the project closing and executed during the project grace period, negatively impacted disbursement. As a result, total disbursement reached 80% at project closing.
Procurement. Given that the entire Government program followed the Bank’s procurement guidelines, the Bank reviewed and approved a large volume of transactions. The initial delays due to lack of planning by the counterpart and long processing time for No Objections by the Bank were addressed and resolved after the first two years of project implementation. Procurement delays remained associated mainly with the scarcity of qualified contractors and remoteness of the work sites.
2.5 Post-completion Operation/Next Phase
Building on the PARSALUD know-how, a follow on GOP investment program, the Programa Nacional de Inversion de Salud (PRONIS) was approved in 2015 and is now operational. In line with the decentralization, PRONIS allows regions to set their own priorities and request funding from the central level. The possibility of a follow-on Bank-funded operation to support this new investment strategy has been discussed. MINSA and PARSALUD presented concrete proposals to MEF and the Bank (including a logic framework for the new operation). Discussions are still ongoing until the political situation stabilizes after the national elections which are taking place between April and June 2016.
