Executive summary
Yüklə 455,2 Kb.
|
Partner Institution
The program’s selection criteria for trainers were simple enough: trainers were to hold at least a master’s degree, preferably a PhD, but most importantly had to belong to promising institutions. The key factor, therefore, was to evaluate the potential of the nominating institutions. WBI had commissioned in FY97 the creation of a set of guidelines for selecting country partners1. The partnership program borrowed from this a very comprehensive system for measuring the capacity of potential partner institutions, which was presented in the project document and which the program was expected to implement. The expected results of the first phase of the program were twofold:
Participants trained during Phase One were to be informed of the partnership program, and they would in turn inform their institution. Phase Two, the delivery of WBI core courses was expected to start when applicant partner institutions sent a letter of intent to the partnership program (see Chart 2). The applicant would then make a proposal describing course content and speakers. The task manager for the course would consider the content of the letter and the proposal, negotiate the financial conditions of the delivery of the course, and select some participants and speakers. The task manager would also be responsible for assisting the partner in the delivery of the course. The partner was expected to adapt the core course designed in Washington to local conditions. It was also expected that the partner institution would help identify participants, and reach an audience that had not previously benefited from WBI core courses. Finally, partner institutions were expected to contribute to WBI’s knowledge base through the development of regional case studies. For WBI, a successful partnership would come with the delivery of a regional core course which possessed the same level of quality, presumably at a lower cost than prior to the establishment of the partnership. 3. The operating context The idea of testing a partnership approach is not new at WBI; several similar programs are underway. For example a health sector program aims at building the capacity of partner institutions, and an environment sector program deals with building the capacity of a network of experts. Other experiences concern partnering with regional institutions. Concerning WBIEP, informal interviews with the task managers show that before the launching of the program, a number of task managers were already delivering regional events, using the facilities of institutions in the developing countries, and at times local speakers. Typically, a newly arrived task manager would obtain from WBI regional coordinators the names of institutions with which to partner. More seasoned task managers would nurture their own network of partners throughout the world. The drawbacks for WBIEP of such a system were significant: when a task manager left the division, there was no guaranteed way to safeguard the efforts that had been invested into developing his/her personal network of partners. There was no centralized information system capturing the strengths and weaknesses of partners in client countries, and no incentive for task managers in Washington to use regional speakers (in fact many WBIEP events were taking place in the United States). Also, the treatment of partners would vary across task managers. The recent policy guidance documents of the World Bank Institute (WBI) and the World Bank at large have put a new emphasis on the development of partnerships. The partnership program was launched in WBIEP at the same time that new courses were developed and new managers hired. It is therefore difficult to construct a clear “before program” situation. An idea of what would have been the situation in the absence of the partnership program can be observed in the regional deliveries of WBIEP core courses early in FY98, before the completion of the two phase cycle of the partnership program. In FY98, WBIEP had a portfolio of five core courses, which were delivered eight times during the fiscal year. Two sessions were held in a client country. Five events were held in Washington, and one in another developed country (Vienna). This indicates that at least a few WBIEP managers had by early 1998 established links with local institutions which were used for the delivery of these five core courses. The role of these institutions, as surmised from the interviews with the task managers in charge of the courses, is presented in box one. Box 1 T MEFMI (a consortium of banks) was identified for the delivery of the course on Economic Growth in Harare (March 23- April 3, 1998). MEFMI was recommended to the task manager by the regional coordinator in WBI. According to the task manager in charge of the course, the input of the partner institution was significant. MEFMI participated in the design of the course, assumed part of the cost, selected participants, identified some presenters (hired by WBIEP), and conducted their own evaluation. In FY98 the course on Economic Growth focused solely on macroeconomic aspects. In FY99 the course was revised to include a poverty reduction orientation. This latter aspect did not interest MEFMI, which was therefore not a candidate for offering the course again in FY99. MEFMI and the African Economic Research Consortium (AERC) were the two partners of WBIEP for the course on Macroeconomic Management that was held in Nairobi in April, 1998. WBI was solely responsible for the design of the course. MEFMI and AERC participated in the selection of participants and MEFMI funded some participants. Out of the fourteen speakers, twelve were World Bank staff, professors from American universities, or individuals from other developed countries. One speaker was a member of AERC, and one speaker was a professor from Bogazici university. he role of local institutions in the delivery of WBIEP core courses in FY98 4. The scope of the evaluation In FY99, WBIEP had a portfolio of five core courses which were all included within the partnership program:
In FY98, the partnerships program had established links with two WBIGF courses (Budgeting Processes and Public Expenditure Management, and Building Knowledge and Expertise in Infrastructure Finance) and one WBIHD course (Social Security and Pension Reform). These three courses are not addressed in this evaluation, which was made at the request of the WBIEP manager. It was decided that the evaluation questions would focus on program implementation in relation to the program’s underlying logic model, i.e. the assumptions upon which the program was based. This evaluation methodology is essential for obtaining information about the future opportunities of the program. The purpose of this analysis is to uncover causal processes to understand why the partnership did or did not work. The more sound the assumptions on which the program is based, the greater the chance that the program will be effective. It was decided that the evaluation would not have a strong focus on cost-benefit aspects of the program for WBI. The reasons are twofold:
The evaluation did, however, address the issue of cost sharing between WBI and partner institutions for the delivery of the core courses. 5. The evaluation questions The evaluation has addressed three major questions:
The evaluation matrix details the more specific sub-questions addressed by this study. It is presented in Appendix I. 6. Methodology The evaluation first reconstructed the logic model of the partnership program. The logic model is a presentation of the logical assumptions and relations among four components of the program as they are described in official documents: the program activities, the type and level of services delivered and the client served, the direct changes in clients as a result of their participation in the program, and the desired results. The first step for the evaluation team was to construct a diagram presenting the flow of the different activities of the partnership program to illustrate how these were expected to induce desired outcomes, considering the assumed (social and behavioral) premises or mechanisms that lie behind them. Data collection centered on how the program activities were implemented, how they were perceived by partners and what intermediate results they generated. The analysis then compared the implementation of the program with the logic model, identified regularities and discrepancies, and suggested recommendations for implementation improvement as well as program redesign. Evaluation data were gathered through four methods:
The evaluation methodology has not distinguished among the countries in which the programs were offered, nor among the five core courses. II The program’s underlying logic The objective of this analysis will be to reconstruct the assumptions (premises) that are embedded in the partnership program. These premises will then be confronted in the following chapters with evidence gathered during the first cycle of the project implementation, to test the validity of the assumptions and identify weak causal links between activities and results. The reconstruction will be presented in two parts. The first set of assumptions addresses the rationale that the sequence of activities proposed by the program (training trainers, consequently selecting among the high quality partner institutions, and jointly delivering WBI core courses in the regions) is an effective way to build sustainable partnerships for the delivery of core courses. This analysis highlights the assumptions of the causal mechanisms underlying the functioning of the partnership itself. It provides an understanding of the potential of the program and also provides a diagnosis of the potential problems for future program improvement. The purpose of this analysis is to uncover the causal processes to understand why the partnership did or did not work. The second set of assumptions focuses more generally on the WBI partnership rationale for the delivery of core courses and the hypothesis that the delivery of a core course in partnership allows for the benefits WBI expects: the maximization of cross-fertilization of experiences and program successes, the combination of strengths of different institutions, the more efficient use of resources, and the exposure of WBI core courses to a wider audience. This reconstruction builds upon the analysis of WBI official documentation and individual interviews with current WBI task managers 1. From training trainers to partnering. The assumptions behind the sequence of program activities The participation of selected participants for WBI core courses is the starting point of the establishment of a partnership for a regional delivery of the course. The partnership is then built through the joint delivery of the first core course. The assumptions (premises) underlying the idea that the successive delivery of these two phases is an effective tool for building a partnership are the following: The training of trainers’ strategy Two hypotheses relate to the selection of participants: Premise A1: Phase One course participants should be trainers. The role of WBI should be to wholesale training activities. Therefore, trainers should be the main targets in Phase One course delivery, as they are most likely to disseminate the content of the WBI course. Premise A2. Reaching out to high quality training institutions worldwide. WBI aims at establishing as many new contacts, with strong institutions, throughout the world as possible, to create a database of potential partners. Two assumptions concern the impact of training trainers: Premise A3. Capacity building of trainers. It is assumed that the core courses will increase the capacity of training institutions by upgrading the skills of trainers by providing them with cutting-edge information. The program makes the assumption that technical skills are what selected trainers need. Premise A4. Multiplying impact through dissemination. Trainers will use new state-of-the-art materials to build courses and curricula. Training trainers will therefore enable WBI to reach larger audiences and to have a greater impact on the ground. WBIEP partnership building strategy Premise B1.Quality assurance. It is assumed by the program that prior training of trainers is a necessary step for the delivery of a core course by a partner institution. More specifically, a limiting factor to quality is the trainers’ technical knowledge. Premise B2: Training instrumental to partnership. Training trainers will ensure that presenters from potential partner institutions will have sufficient knowledge to deliver the course. It will also enable task managers to screen the most interesting individuals from the lot of trainees, and provide a face to face relationship that will foster the development of the partnership. Premise B3. Informing training institutions of partnering possibilities. Trainers participating in a core course are informed about the opportunity to deliver an adapted version of the WBI core course in-house. It is assumed that trainers are a good point of entry in the institution for the delivery of a core course. Premise B4. The partnership program serves as an essential clearinghouse of the proposals sent by partner institutions. It is assumed that some training institutions, informed of the program by their trainers, formulate a proposal for the joint-delivery of a WBI core course, and submit it to the partnership program. Premise B5. Number of partner institutions. “The project should focus on a small number of institutions worldwide (i.e. 2 in MENA, 2 in ECA, 2 in Africa, 2 in SEA, 3 in LAC, and 1 in SA) to maximize the overall impact. Increasing the number of partner institutions should be gradual and based on experience, best practice, demand requests to enter new markets or new products reaching larger audiences. This gradual process ensures that the project is manageable, cost effective while maintaining quality control and allowing for competition.”2 Premise B6. Selection of training partner institutions. Once WBIEP has received the institutions’ proposals for joint delivery of core courses, WBIEP singles out the training institutions which best suit its educational, organizational, and financial requirements. The selection builds upon the following criteria:
Premise B7. Collaboration on the educational and organizational aspects of the training. Once a common understanding has been reached, the partners begin to collaborate for the actual organization and implementation of the joint program. Consultation develops specifically on the following issues:
Premise B8. Joint delivery of the course. The actual delivery of the course takes place on-site within the partner institution’ s facilities, with the contribution of local presenters and regional case studies. Premise B9. Methods of Teaching: Traditional vs. Distance Education. “A mix of both traditional education, where the trainer is present in the classroom, and distance education is ideal. However, the criteria for selection of institutions should not exclude those without infrastructure for distance education but value the ones that provide these kinds of facilities.”3 2. Why deliver courses with partner institutions? The following set of assumptions focuses on the rationale that the delivery of core courses in partnership allows for all the benefits WBIEP expects for itself and for the partner institution: the maximization of cross-fertilization of experiences and program successes, the combination of strengths of different institutions, the more efficient use of resources, and the exposure of WBIEP core courses to a wider audience. The premises underlying this idea are the following: Premise C1. Co-organization for Comparative Advantage. It is assumed that each partner contributes to the delivery of the course, in the area for which it has a comparative advantage. The process is bi-directional and iterative. WBIEP remains responsible for the overall quality of the course; the comparative advantages of the partner institutions are believed to be with the selection of local participants and speakers, the adaptation of the course to regional conditions, the provision of infrastructure, the logistics, and the obtainment of presenters at competitive prices. Premise C2. Mutual Intellectual Contributions for the delivery of the course. It is assumed that WBI should provide all the state-of-the-art course materials, whereas the partner institutions should integrate and adapt them with local and regional case studies, articles, and the presentations of the local trainers and speakers. Premise C3. Mutual Exchange of Knowledge between institutions. Working in partnership for the delivery of core courses is expected to bring new knowledge into the Bank and provide opportunities for clients to learn from each other’s best practices. Premise C4. Cost-sharing/Cost-recovery. “WBI, where appropriate should emphasize cost recovery as a practical market for programs to increase participant ownership of the program and for revenue enhancement. WBI’s partner should be committed to the partnership by pledging its own resources (funds, time, expertise, existing knowledge products and services) and through mutual commitments of resources and mutually agreed programs of activity. Pricing and charging fees to joint programs has the beneficiary effect of identifying what is being demanded by the final consumer. These revenues could help WBIEP subsidize more needed institution in poorer regions such as Africa and Central Asia.”4 Premise C5. Potential for follow-up. Other activities may be organized and shared together with the partners in the coming years. “The potential success and sustainability of the partnership program arrangement will be greater if the demand side for training is actively engaged and linked with supply side: the needs of clients should be the starting point for getting training providers involved. Finally, the programs being demand driven will ensure continuity in the coming years.”5 III Empirical evidence from Phase One: training of trainers FY98 marked the first round of the training of trainers, with a view to building partnerships for the joint delivery of a core course. We will first consider how the program was implemented and the adjustments that had to be made to the operational mode presented in the first chapter of this report. We will then consider how the implemented program matched against the assumptions (premises) identified above. 1. The selection of institutions and participants As a reminder, the two assumptions related to the selection of participants for Phase One were that the partnership program was to select trainers, and use the selection process to reach out to high quality training institutions worldwide. The selection criteria The selection process that had been originally presented in the project document was modified and simplified. In practice, task managers and the program manager said they applied the following selection criteria. Trainers selected had to present the following characteristics:
Training institution partners had to present the following characteristics:
According to the program manager, the selection process was conducted through peer review (task managers, project manager, division manager). However there is no written record of the selection process nor the grounds on which decisions were taken. Therefore, there exists no documentation concerning decision making by which we can judge what respective weight was assigned to the above criteria for the selection of participants. The nominating process The evaluation team was not able to fully reconstruct the nominating procedure and check how many institutions had been contacted by WBIEP. The database of the partnership program shows that 100 institutions nominated 313 trainers to attend one of the five core courses WBIEP selected for the partnership program. Chart 3 presents the geographical distribution of nominating institutions across the six regions defined by the World Bank. We can see that institutions belonging to the Mediterranean and North Africa (MENA) region represent 34% of the total, the Africa (AFR) region and the Latin America and Caribbean (LAC) region each account for 21% of the nominating institutions, Europe and Central Asia (ECA) account for 15%, and the South Asia region (SAS) and the Eastern Asia and Pacific region (EAP) are the least represented (around 5% each). Chart 3 Regional Distribution of Nominating Institutions 
The selection pressure 141 participants belonging to 60 institutions were finally invited. Chart 4 represents the division of participants across the six regions of Bank Operations. We can see that the LAC and the Africa regions are the most represented (they account for 32% and 23% of the participants respectively), MENA and ECA come next (18% and 15% of participants), and the SAS and the EAP regions are the least represented (6% of participants). Chart 4 Regional Distribution of Participants
The selection pressure on participants (number of nominated candidates excluded, divided by the total of nominated candidates) varies across the regions, the countries and the institutions, thereby suggesting an active role of WBI in selecting desirable candidates. To illustrate this point, let us consider the Africa region. For this region, the selection pressure on participants was highest with only 31% of the nominated participants finally retained (average 55%). WBI invited 32 trainers from 13 African countries, but 60% of these participants came from only three countries: Zimbabwe (nine participants), Ethiopia (six participants), and Senegal (four participants). Furthermore, three institutions account for more than half of the candidates finally selected: the Economic Commission for Africa (Economic Commission for Africa, seven participants), the Zimbabwe Institute of Public Administration and Management (ZIPAM, seven participants) and the Banque Centrale des Etats d’Afrique de l’Ouest (BCEAO, four participants). This emphasis placed on a few countries and a few institutions is observable also for the EAP, SAS, ECA and LAC regions. The exception was the MENA region, where there was a strong collaboration with the Palestinian Authority (half of the 25 participants), with no particular focus on specific institutions by WBI. The reader will find additional details on the selection process across regions in appendix II of this report. An emphasis placed on a few countries and institutions The analysis of the data demonstrates that WBI focused narrowly on nine countries (out of 40), which were able to send six participants or more, while the average number of participants per country was 3.5 (see Chart 5). These nine countries together account for 55% (78) of the participants registered in WBI courses. They are:
Chart 5 Geographical Distribution of Participants Pakistan: 5 2 other countries: 3 Total participants=8 
Zimbabwe: 9 Ethiopia: 6 Senegal: 4 10 other countries: 13 Total participants=32 Palestinian: 12 Egypt: 6 5 other countries: 7 Total participants=25 China: 4
Philippines: 4 Total participants=8 Brazil: 13 Colombia: 12 Venezuela: 7 Mexico: 6 Barbados: 5 3 other countries: 4 Total participants=47 Czech: 7 Uzbekistan: 5 Poland: 4 4 other countries: 5 Total participants=21 Likewise the selection process resulted in an emphasis placed on 14 favorite institutions (out of 60), which were able to send more than four participants to a WBI core course, when the average per institution was 2.3 (see Chart 6). Together, these 14 institutions account for 55% (78) of the people trained in FY98. As we can see in the following list, all 14 institutions favored by WBI were either universities, regional institutions, or research institutes. They are:
Institutional origin of participants Universities and research institutes make up the bulk of the institutions which sent participants to the core courses in FY98 (40% and 28% of the trainees, respectively). However management schools (8% of participants), governments (14%), intergovernmental organizations (7%) and the private sector (3%) were also represented. Trainers vs. non-trainers A questionnaire was sent to all FY98 participants registered in the partnership program database. The response rate was 55.8%. One question addressed the participants’ current role in their organization. Forty six percent of respondents described themselves as trainers or professors, 20% answered that they were researchers, 27% were working as directors or at another management position, and 7% were private sector or development specialists. Less than fifty percent of respondents classified themselves as the category targeted by the program. Of course participants from categories other than trainers or professors may also occasionally deliver courses, but we have no measure of the extent to which they do. Another question addressed the motivation of participants in attending the courses. Interviewees were asked to answer the last part of the questionnaire concerning their use of the course only if they had attended it with the view to enhance their training skills. Out of the 77 respondents to the questionnaire, 62 (80%) participants answered the second part and 15 (20%) did not answer it, which suggests that 80% of participants to the core courses were interested in enhancing their training skills (note that this does not necessarily translate into an intention to use these skills in teaching). In conclusion, although 80% of respondents had attended the workshops to enhance their training skills, only 46% percent of respondents described themselves as trainers or professors. Conclusions for the phase of selection Premise A: participants to Phase One courses should be trainers. Less than 50% of the participants listed for the partnership program in FY98 described themselves as trainers or professors. An additional 30% of the participants were interested in enhancing their training skills and a proportion may be occasional trainers, while 20% of the participants did not attend WBIEP core courses to enhance their training skills. We may therefore conclude that in FY98 the targeting of the program on trainers was incomplete. Premise A2. Reaching out to good quality training institutions worldwide, so as to create a database of potential partners. Evidence for the first year of the program implementation shows the following results:
2. The outcomes of the training The outcomes of the training of trainers phase of the program relied on three hypotheses: one was that the level of knowledge of participants would be increased, the second was that the participants would disseminate the new knowledge they had acquired in delivering training sessions, and the third was that the identification of qualified trainers by task managers would foster the development of the partnership. The data for this section of the analysis have been gathered with a tracer questionnaire (see Appendix III) sent to all participants in FY98 courses. As we have seen above, a significant proportion of the participants to the core courses selected for the delivery of the partnership program did not describe themselves as trainers or professors. We have therefore distinguished, where feasible, between the impact of the training for the populations of trainers and non trainers. The two groups have comparable sizes (thirty three trainers and forty non trainers, there were three non respondents). Did the core courses in FY98 increase participants’ technical knowledge? As there was no level II evaluation conducted at the end of the core courses to measure the participants’ increase in knowledge, it was not possible to determine the actual learning impact of the training sessions. Did participants state that the training in FY98 was useful to them or to their organization? The questionnaire sent to all participants in the FY98 courses gathered information about whether they believed that they had learned from the training, and to what extent they considered it a useful experience for themselves and for their organizations. A worthwhile personal and professional experience was reported, but some participants, especially trainers, have doubts about the worth of the training for their work or their organization. A five point scale, ranging from 1=minimum to 5=maximum was used for answering questions regarding the usefulness of the training. The results are summarized on Charts 7 to 9, and detailed in Appendix IV. We have distinguished between the results for the entire population of 77 respondents, the trainer sub-set, and the non trainer sub-set. Overall, the participants felt that the seminar was a worthwhile experience for them personally and professionally. The proportion of respondents giving high ratings (% of 4 and 5) was 83% and 81% respectively, slightly below the WBI benchmark of 85%. Results from two indicators suggest that participant interest in the courses was more theoretical than practical. One indicator was the practical contribution of the seminars to the understanding of policy options (Chart 7), and the other was the relevance of the seminars to the participants’ functions (Chart 8). These two items are a measure of how well the seminars are adapted to participants’ needs. The proportion of respondents giving a rating of 4 or 5, 63% and 67.6% respectively, were markedly below the benchmark for WBI (85%). In general, respondents felt that the seminar they attended was less worthwhile from their organizational perspective than it was from a personal point of view (see Chart 9). For this item, the proportion of dissatisfied participants (participants giving a rating of 1 or 2) is 14%. This can be problematic, as a financial contribution was requested from the organizations. Chart 7 To what extent did the seminar strengthen participants’ understanding of policy issues? (1=minimum; 5=maximum)
Chart 8 Relevance of the seminar to participants’ work (1=minimum; 5=maximum)
Participants who do not describe themselves as trainers or professors were more satisfied by the core courses. Non-trainers gave high grades to the training for personal and professional satisfaction (more than 85% of them gave a mark of 4 or 5 to these two questions). The non-trainer subgroup gave the highest scores to the questions related to the relevance of the training to their work (73% marked 4 or 5), the usefulness of the training for their organization (76% marked 4 or 5) and the strengthening of their understanding of policy issues (75% marked 4 or 5). Chart 9 Relevance of the seminar to participants’ organizations (1=minimum; 5=maximum) 
The results for the subset of trainers are more problematic. Trainers reported that the training was a positive personal and professional experience. But 15% of them gave a low rate of 1 or 2 to the question relating to their understanding of policy issues. Trainers gave lower ratings than non-trainers for the relevance of the training to their work (60% rating 4 or 5). And almost 19% of the trainers had serious doubts (ratings of 1 or 2) about the usefulness of the training for their organization (see chart 9 above). Yüklə 455,2 Kb. Dostları ilə paylaş:
|
BI accredits strongest partner institutions;
BI proposes arrangement concerning cost sharing and responsibility sharing. WBI selects some of the trainers, and selects international participants to the regional course;
