Ilo evaluation
Main Evaluation Steps and Activities
Yüklə 2,52 Mb.
|
- Bu səhifədəki naviqasiya:
- 3.Main Evaluation Findings
- 3.1Follow up on recommendations Final Evaluation SCORE Phase I
- 3.1.1Recommendations at the global component level
- 3.1.2Recommendations that apply to all SCORE countries
2.4Main Evaluation Steps and Activities
Initial intake, interviews at ILO headquarters in Geneva and lecture of documents as input for the inception report outlining the evaluation approach and methods, including the evaluation matrix, a final work plan, briefing notes and questionnaires.
Review of project documents at the global and country levels together with an analysis of available data in the SCORE M&E system.
The evaluation team visited seven countries. The visit schedule to the countries was as follows: table : Schedule country visits
During the field work the evaluation team visited 16 companies for meetings with owners, managers and workers. In all these companies, enterprise walk-throughs were conducted. In total 126 meetings were held, both individual and group interviews, with SCORE teams as well as representatives from the ILO country offices, government, trade unions, employers’ organizations, trainers, companies and others.
At the end of the country visits, the evaluators organized a debriefing and validation meeting with national ILO offices to share findings and check missed elements in the data-collection process. Preliminary conclusions and recommendations were also discussed with ILO staff in Geneva.
The draft evaluation report with findings and recommendations was presented to the evaluation manager in January 2016. The final report and evaluation summary were submitted on 24 February 2016. 3.Main Evaluation FindingsThe findings presented in this chapter are a synthesis of many more specific findings that were obtained through extensive fieldwork in seven countries where SCORE is implemented. For these more specific findings the reader is referred to Annex 3 - 9, where specific country reports for all SCORE countries are presented. 3.1Follow up on recommendations Final Evaluation SCORE Phase IAt the end of the previous SCORE Phase I project period an external final evaluation was realized. The final evaluation report (December 2012) presented a series of conclusions and recommendations at the global level as well as conclusions and recommendations that applied to all countries with a SCORE programme. In this section the evaluators analyze to which extent the recommendations of the final evaluation of SCORE Phase I have been implemented. 3.1.1Recommendations at the global component level
With advance funding from NORAD, Phase I was extended to December 2013. Phase II started in November 2013, which allowed for a smooth transition between the two phases. This provided the possibility for a longer-term strategy and approach towards product branding and marketing by finalizing the first project-pilot phases and generating experiences that feed into knowledge sharing and communication efforts in SCORE countries and at the global level.
With the formal start of a five-year Phase II SCORE programme in November 2013 and factual start of operations in January 2014, this recommendation is not directly relevant anymore. However, it should be noted that in Phase II, the SCORE implementation in China, Indonesia, Vietnam and South Africa was designed only for four years until the end of the 2016, while the other SCORE countries continue implementation until the end of 2017. At the time of this Mid-Term Evaluation there are clear indications that the four-year time frame from January 2013 until the end of 2016 is too short to ensure that these SCORE countries can achieve sustainable continuation. Particularly in Vietnam, where SCORE only started in 2012, the time frame is too short for ILO to transfer the SCORE project to partners (see further below).
The SCORE teams both at central and country level have clearly followed up on this recommendation and have given much attention to SCORE communication and marketing. This is done through case studies and testimonials on SCORE websites and in the form of brochures and booklets. Several of the country programmes use social media like twitter and face book to different degrees. The marketing and communication is organized in collaboration with private sector branch organizations and associations, although it seems that ILO itself remains prominent in these communication efforts. The marketing and communication around SCORE seems to focus on the interest of the SME owners and managers to increase competiveness and productivity. The sales argument is that participation in SCORE will generate material and financial benefits to the SMEs. Although this is a clear strategy, this Mid-Term Evaluation detects that it is not sufficient to effectively roll out SCORE training and support over more SMEs in the SCORE countries. One of the recommendations of this evaluation report is to add as a selling point that integration in supply chains increasingly requires compliance with international labour standards and certification of this compliance. Participation in SCORE can help SMEs to prepare for this certification.
The SCORE team has made a lot of effort in developing gender strategies and mainstreaming it in the SCORE services. For all SCORE countries gender-plans were developed. The increased attention has had a clear effect on the participation of women in trainers’ teams and among SCORE participating companies. Only in India, despite efforts to recruit female trainers, their participation continues to be on the low side. This is caused mainly by the fact that there are hardly any female engineers in India. A special effort is made by SCORE to involve female-owned enterprises and gender aspects are addressed in the training and consultations. Several enterprise improvement-plans focus on specific demands and interests of female workers. It nevertheless remains difficult to measure to which extent women and men benefit equally or differently from SCORE training and consultations.
The global SCORE team has worked hard on improving and clarifying the results-chain of SCORE and for all countries a specific theory of change and results-chain were developed and visualized. The project also underwent an audit against the DCED results measurement standard, which was passed with 87% compliance. Information on the ILO papyrus website, regional coordinating meetings and exchanges like a visit of SCORE India to Indonesia to learn more about the collaboration in Indonesia with the government, are used to share knowledge. Different experiences in different parts of the world offer interesting learning opportunities for SCORE teams. A challenge in the SCORE project is that the M&E system is very much geared towards aggregating and synthesing data on some Key Performance Indicators (KPIs). On the one hand this enables more concise reporting on SCORE at the central level, but on the other hand the practical experiences in reporting on the performance indicators show that these are not always relevant for the implementation of the SCORE project in specific sectors and at the company level. This evaluation will show in sections 3.3.3 and 3.5 that a significant part of the information generated on the KPIs is not very reliable and not directly relevant to changes at the company level. Therefore, the follow-up on this recommendation to increase M&E efforts has been considerable, but it is not evident if these M&E efforts are actually useful in generating the right knowledge to steer the project.
The evaluators have not seen specific follow-up on this recommendation. But in general the SCORE trainers integrate KPIs in their monitoring and follow-up on enterprise improvement plans. The SCORE modules and training methods from the start of the SCORE project were already designed along the WISE methodology, so this specific recommendation seems to be somewhat redundant. A challenge remains the aspect that was already mentioned under the previous recommendation. Not all KPIs are directly and immediately relevant for specific enterprise improvements and therefore don’t always make sense to SME owners, managers and workers.
There has not been any central steering on common terminology in communication and marketing around SCORE. In different countries this is done differently. At the central level the SCORE programme document introduces common concepts and common language, but at the country level this is adapted and fine-tune and sometimes practical developments require more pragmatic approaches instead of adhering to a common terminology at the global level. The most obvious example of limitations in using a common terminology is the concept of SMEs. Although there is a common central definition of SMEs - between 50 and 250 employees - these thresholds are dealt with differently in different countries and sectors.
SCORE has followed this recommendation. A number of countries (e.g. China, India and Colombia) have been engaging lead buyers in national and international supply chains. Other countries (Indonesia) so far have focused more on the involvement of government institutions. In most countries, there are still challenges in involving lead buyers. The potential of working along the value chain and international supply chains as partnership approach, however, is at the time of this Mid-Term Evaluation widely recognized by stakeholders in different SCORE countries. 3.1.2Recommendations that apply to all SCORE countries
This recommendation is followed in all SCORE countries, but it does not always lead to sustainable partnerships as sometimes institutions lose interest in the SCORE approach. For example, in Ghana one of the partners announced its intention to become the national centre for the implementation of SCORE in the country and even signed a MoU. But the organization lost interest and did not fulfil its part of the agreement. Bottlenecks in terms of allocation of government budget for capacity development of the SME sector were found in several SCORE countries. One other bottleneck that was not recognized in this recommendation is the fact that many support services to SMEs provided by the government, are under strict government curriculum control and cannot be offered against payment, while SCORE is explicitly working towards sustainability of SCORE training and services by increasing the share of cost-recovery of SCORE training by participating SMEs. These differences in vision on sustainability create clear challenges in working with public institutions in SCORE training provisions (e.g. in Colombia, South Africa and Indonesia).
The intention of this recommendation is not entirely clear. In SCORE clear targets are set for the number of SMEs and trainings at a half-yearly base in each country. Against what other indicators these targets should be benchmarked, is not specified in the recommendation. In each country a different pathway is followed for training and service delivery and outreach among SMEs. Furthermore, strategies for offering one or more modules and sometimes offering combined modules are different in each of the SCORE countries. This illustrates that differences in the country context do not allow a single and uniform approach in SCORE training delivery and this makes benchmarking of targets also not a very useful exercise.
Sustainability is mentioned as core objective in the project document for SCORE Phase II. For all countries targets are set to increase sustainability and outreach of SCORE training, but in all countries the challenge of offering SCORE as a service with full cost-recovery remains and in none of the countries incremental cost-recovery moves sufficiently towards a sustainable supply of this service. An evaluation of the SCORE program in India in 2013 mentioned the unwillingness of private enterprises to undertake actions that will yield public benefits. In such cases subsidies may be needed.6 In this Mid-Term Evaluation we also argue that to increase sustainability SCORE should recognize the pluriformity of the SME sector and introduce specific cost-recovery and payment mechanisms as well as partnership arrangements for specific categories of SMEs.
All country teams have worked on developing country specific and appropriate communication and information around SCORE. Additional instruments were developed at the central level to assist country-level stakeholders to use and exchange information, such as a Papyrus website on SCORE. This website is mostly used as a document storage facility and updated with training materials and monthly, quarterly and bi-annual reports.
This recommendation has been implemented. As could be observed by the evaluators during company visits, SMEs receive a certificate of participation in the SCORE modules.
The evaluators in this Mid-Term Evaluation have not observed that planning of SCORE training courses was not appropriate to companies. Planning of SCORE modules is done together with industry associations and SMEs and it appears to be a normal practice that training is done at appropriate moments. Yüklə 2,52 Mb. Dostları ilə paylaş:
|