Ilo evaluation

2.Mid Term Evaluation of SCORE Phase II

2.1Evaluation Background

This review is a mid-term evaluation of the second phase of the SCORE programme. It is therefore important to balance looking back with looking forward. The evaluation tries to generate insights and practical suggestions to feed into processes to improve the implementation of the programme. The objectives of the evaluation are the following.

• 
  Independently assess the progress of SCORE in Phase II against the logframe.
  
• 
  Assess the degree to which the recommendations from the final evaluation in Phase I and the evaluations of SCORE India and South Africa were implemented.
  
• 
  Inform the ILO on whether the current project strategy is working and provide recommendations on what could be changed to increase the likelihood that the project reaches its objectives.
  
• 
  Identify good practices and lessons learned that would contribute to learning and knowledge development of the ILO and stakeholders.
  

The evaluation covers the period from December 2012 until November 2015 and evaluates the SCORE components in Colombia, Ghana, South Africa, India, Indonesia, Vietnam, China and the global component. These countries were visited between October and the first week of December 2015. The final evaluation report was submitted in February 2016. The intended audience includes the SCORE staff, ILO country offices and other field and headquarter staff; the donors SECO and NORAD; and tripartite members of the global and national advisory committees and partner organizations in the evaluated countries. A summary of the evaluation will also be made available publicly through the websites of SCORE and the ILO Evaluation Office.

2.2Evaluation Criteria and Questions

The evaluation addresses a series of questions related to relevance, intervention design, effectiveness, efficiency, sustainability and management arrangements. Some key questions outlined in the terms of reference are the following (see Annex 13 for the ToR with a full overview of the evaluation criteria and related questions).

1. 
   Relevance and strategic fit of the intervention
   

• 
  Are the objectives of Phase II consistent with beneficiaries’ requirements, country needs and policies of partners and donors?
  
• 
  Is the programme linked to national and ILO’s development frameworks?
  

2. 
   Validity of intervention design
   

• 
  Are the project strategy and objectives appropriate for achieving results?
  
• 
  Did project design adequately consider the gender dimension of interventions?
  
• 
  What else can the programme do to promote trade unions in SMEs?
  

3. 
   Intervention progress and effectiveness
   

• 
  Is the programme on track in delivering its outputs in all countries?
  
• 
  To what extent has it so far achieved its objectives, including cost-recovery?
  
• 
  Does it lead to the desired impact on enterprises and their workforce?
  
• 
  How far has the capacity of partner organizations been built in relation to the delivery of outputs and objectives?
  
• 
  What obstacles did the programme encounter and what corrective action is needed to achieve the objectives?
  

4. 
   Efficiency of resource use
   

• 
  Does the project make efficient use of its financial and human resources?
  

5. 
   Effectiveness of management arrangements
   

• 
  Are time frames and work plans respected?
  
• 
  Is the management structure effective?
  
• 
  Are tripartite advisory committees functioning?
  
• 
  Is the programme systematically monitoring, documenting and communicating results? Is the monitoring system practical, useful and cost effective?
  

6. 
   Impact orientation and sustainability of the intervention
   

• 
  How effectively has the programme built national ownership and capacity of people and institutions? Are partners willing and able to continue the programme after funding ends?
  
• 
  Are there business models applied in countries that seem more promising to reach financial sustainability?
  
• 
  Are the gender-related outcomes likely to be sustainable?
  
• 
  Has the project reached sufficient scale and depth to justify the investment? Are the approach and results likely to be upscaled or replicated?
  

The list of specific research questions is quite extensive. In consultation with the SCORE team at ILO headquarters it was therefore decided to focus the mid-term evaluation on the effectiveness of programme implementation, management arrangements and the perspectives for sustainability of the SCORE training delivery. Less focus is on the relevance and strategic fit and efficiency of resource use, because these aspects have been looked at in previous evaluations and will also feature more prominently in the final evaluation. The evaluation criteria and prioritized elements are brought together in an evaluation matrix that served as a structuring tool for the design of the research instruments and for the process of analysis and reporting. The evaluation matrix is presented in Annex 12 of this report.

2.3Evaluation Approach and Methodology

A team of three international consultants and two national consultants conducted the evaluation. They were assisted by a research assistant, based in Vietnam and thus could also provide assistance in the fieldwork in this country. In Ghana, India and China national consultants were hired to assist the evaluation process and provide a reality check with regard to the country-specific context in which the programme operates. Unfortunately, the consultant in China had to withdraw. Due to the extensive experience of the team leader in Colombia, no national consultant was proposed for this country. In Indonesia, Vietnam and China interviews were undertaken involving the use of an interpreter.

2.3.1Methodologies used

The evaluation draws on the use of various sources of information, including policy documents, project data, interviews, factory visits and written surveys. The following methodologies were used during the evaluation.

1. 
   Document review
   

The evaluation team reviewed project documents on both global and national levels, including work plans, progress reports, training materials and reports from previous evaluations.

2. 
   Data review
   

Existing quantitative and qualitative data from the programme’s M&E system were reviewed to track the programme’s progress in achieving its objectives.

3. 
   Semi-structured interviews
   

Interviews were conducted in a semi-structured way to ensure objectivity and consistency in interviews in the different countries. For that purpose the evaluation team developed interview checklists during the inception phase. These were based on the specific research questions from the evaluation matrix. The evaluation team developed separate checklists for the following stakeholder groups: ILO programme officers, service providers that provide training to beneficiaries, owners and managers of companies, workers and partners such as employers’ organizations, trade unions, government and donors. For interviews at the global level no format was used.

4. 
   Group interviews
   

Besides individual interviews, the evaluation team conducted group interviews with groups of managers, workers and trainers. This enabled the discussion and crosschecking of information between respondents. The interview checklists mentioned above were used to structure the discussions.

5. 
   Short surveys
   

At the end of individual and group interviews with beneficiaries and stakeholders respondents received a short hard-copy survey to provide assessment of developments and changes as a result of SCORE interventions in the period of implementation of the project. These surveys complement the information from interviews and also generate data on opinions and appreciations of SCORE for comparative purposes.

6. 
   Factory visits and walk-throughs
   

During each country visit, the evaluators tried to visit at least two SMEs. They were requested to organize meetings with the owner, two or more managers and six or more workers (with a minimum of two female workers) who had participated in SCORE modules. If present in the company, a meeting was also requested with trade union representatives. Interviews were combined with walk throughs for which the evaluators used a checklist for the observation of company actions and measures as follow-up to SCORE modules, such as clear and visible instructions for safety and emergency procedures, systems to collect waste and gender specific provisions in the company.

7. 
   Gender assessment
   

The evaluation used two separate gender-checklists to systematize findings on gender. A first checklist was used at the SME level to check gender aspects in the follow-up of SCORE training and consultations. At the end of each country visit, a second gender-checklist was used to analyze gender aspects at the level of the SCORE programme in that particular country.

2.3.2Limitations and sources of bias in data-collection process

During the evaluation process the evaluators have encountered the following challenges:

• 
  The time for visits to companies was limited. Although it was not the focus of this mid-term review to look at outcomes and impacts at the enterprise level, these company visits were important to better understand the intervention logic of SCORE and also to identify bottlenecks and challenges in achieving sustainability. In total 16 companies could be visited. This number is too low to be representative for the total number of 765 enterprises that have been benefiting from SCORE to date. Nonetheless, the impressions of the enterprise-visits and interviews with owners, managers and workers have contributed to the overall analysis in this evaluation.
  
• 
  ILO and implementing partners of SCORE selected the visited SMEs. All enterprises selected were still applying SCORE methodologies and had active Enterprise Improvement Teams (EITs). This means that evaluators have seen best practices of SCORE and no enterprises where SCORE training was not completed or where after the training no continuation was given to SCORE. The visits to SMEs should therefore be seen as an impression of the potential effects that SCORE can have in case companies are sufficiently committed to the programme and also managed to remain in business over time. An impact evaluation would be required to obtain more information on the impacts at company level.
  
• 
  Heavy flooding in Chennai influenced the field visit to India. It was not possible to visit the region and to talk with different stakeholders and conduct enterprise visits in this region, which has been the core region for SCORE interventions in India. Instead an alternative visit closer to Delhi was arranged, but it was not possible to visit more than one company.
  
• 
  The M&E on-line platform is an interactive system that can be updated on a day-to-day basis. While conducting the analysis on the on-line platform, data kept changing, which made the analysis quite challenging. In the end the evaluators have downloaded and analysed the data on the on-line platform on December 20, 2015, assuming that around the Christmas and New Years recess limited or no information would be added. However, it is possible that data in this report might divert slightly from data on the on-line platform in case after 20 December 2015 SCORE teams and trainers still have updated information.