Ilo evaluation

The data in the table show that reporting on most indicators is not very reliable as we see percentages that are both positive and negative and this is not to be expected except for wage development. The final two indicators seem to suggest that effects on wages are very limited. And most importantly the effects on cost savings are very modest with an average of 359 USD per company. This is much lower than the cost-price of SCORE training modules.

At the highest level, outcomes were reported on job-creation effects. The results (until 20 December 2015) are presented in the table below:

Enterprises Trained	95
# enterprises for which data is available	77
% Enterprises providing basic reporting	81%
Job creation
Nr. of jobs created in enterprises trained	530
Nr. of jobs lost in enterprises trained	-144
Net change	386
Changes in employment status
Increase in casual workers	1248
Decrease in casual workers	-13
Net change	1235

Effects of SCORE on job-creation are significant, but it is also remarkable that part of the employment created is not in fixed jobs. Particularly casual jobs are created according to the ME& platform. This leaves a challenge to improve status of jobs in Colombia.

During the evaluation visit the evaluators have visited three SME’s to confirm achievement of outcomes in enterprise walkthroughs. The changes observed at SME level are presented in the table below. The average scores are given on a three-point scale (1=low, 2=sufficient, 3=good)

SME Walk Through Checklist	Score
1. visibility CSR expressions	2,67
2. Visibility certificates (ISO etc.)	2,00
3. Overall cleanliness	3,00
4. Clear instructions workers safety	3,00
5. Visible OSH aspects	2,67
6. Signs energy efficiency	2,33
7. Gender specific provisions	2,67
8. Waste management systems	2,33
9. Facilities worker’s social wellbeing	2,67

Effects of SCORE are most visible in cleanliness of companies (results of 5-S) and in OSH measures introduced at the work floor. On waste management the evaluators have not seen many changes.

Appreciation of workers on the application of SCORE in their companies are presented below on a five-point scale (1=not at all/very low, 2=low; 3=sufficient, 4=high, 5=very high):

Survey workers in SME’s benefiting from SCORE	Score
1. Awareness ILO/SCORE	4,25
2. Participation in SCORE	4,38
3. Quality SCORE	4,38
4. Performance trainers	4,50
5. Effects on Management-Workers dialogue	4,50
6. Changes in company after SCORE	4,25
7. Effects on labour conditions	4,38
8. Effects on worker’s organization & unionization	4,14
9. Effects on women participation	4,38

Workers are most positive about the effects of SCORE on their knowledge and experience and also on improvements in the workplace. Most critical are workers about the effects of SCORE to create institutional forms of worker’s organization, which reflects the historically difficult trade union rights situation in Colombia.

2. Opinions and Appreciations of SCORE by partners and beneficiaries

2.1 SCORE team

• 
  The Score team in Colombia is generally satisfied with progress of SCORE, particularly when considering the very slow start of phase II in Colombia. The approval of the country strategy for SCORE took a long time;
  
• 
  SCORE was mostly inactive for the first one-and-a-half year of phase II and many of the results obtained in phase were lost;
  
• 
  The SCORE team is convinced it will reach the targets of SME’s to be trained by SCORE towards the end of phase II, but it is sceptical about the sustainability perspectives;
  
• 
  Most perspective is seen in developing a combined SCORE module 1 and 5 training to achieve better cost-efficiency and pricing. Module 5 also presents interesting options for cooperation with insurance companies to pick up part of the funding for Module 5 training by anticipated savings in OSH related accidents. In a proposal with ANDI to INNpulsa modules 1 and 3 are considered and a second proposal with ACOPI module 1 and 2 are included. So it also means that modules 2 and 3 are also considered but in more specific project proposals to an external funding programme of the Ministry of Commerce and Industry;
  
• 
  The local SCORE team is satisfied with support from ILO HEADQUARTERS and systems are generally working. Only with respect with M&E there is criticism on the relevance and user-friendliness of several elements of the SCORE M&E platform and system.
  

A small survey among SCORE staff shows the following appreciations on a five-point scale:

	Score
1. Integration SCORE in ILO policies & plans	4,00
2. Appreciation SCORE by Government (as perceived by SCORE officers)	3,33
3. Appreciation SCORE by Employers	5,00
4. Appreciation SCORE by Trade Unions	2,67
5. Appreciation technical support from central SCORE team by national teams	4,33
6. Perspectives future sustainability	4,00
7. Change behaviour SME’s after SCORE	4,00
8. Gender equity in SME's after SCORE	2,50

The survey responses are in line with the interview findings with the different stakeholder groups. SCORE has excellent relations with employers but relations with trade unions are not very strong and constructive. The SCORE team also still sees challenges in the area of gender equity, although in terms of participation there are no challenges. The challenges are in the area of ensuring that women can benefit equally from improvements in companies.

2.2 SCORE trainer’s and SCORE training service organizations

A small survey among SCORE trainers shows the following appreciations on a five-point scale:

	Score
1. Growth SME demand for SCORE	2,67
2. Tailoring SCORE to SME needs	3,83
3. Appreciation SCORE by SME owners	4,00
4. Appreciation SCORE by SME workers	4,42
5. Quality SCORE modules & materials	4,17
6. Technical support from HEADQUARTERS SCORE team	4,17
7. Perspectives future sustainability	2,82
8. Change behaviour SME’s after SCORE	4,08
9. Gender equity in SME's after SCORE	3,55
10. Willingness & capacity SME’s to pay for SCORE	2,11

The trainers in Colombia are slightly more critical on the quality and effects of the Score project and effects than in other SCORE countries. Particularly on capacity and willingness to pay for SCORE and related to this perspectives for sustainability trainers in Colombia are not optimistic. This is in contrast with the relatively high rates of cost recovery (but still below 50%) in Colombia as compared to other countries. Trainers also refer to the fact the Government provides free training to SME’s through SENA and this limits possibilities for SCORE to reach cost-recovery of its training provision.

2.3 Government

• 
  The Ministry of Labour is aware of the SCORE project of ILO, but is not very familiar with it. Contacts were established around the SCORE project, but these are not yet sufficiently followed;
  
• 
  Ministry is happy with support to SME sector and particularly on labour relations in this sector;
  
• 
  Awareness of SCORE initiative to try to establish cooperation with Insurance Companies to finance the supply of combined module 1 and 5 (OSH) by the Ministry is limited. More follow-up by SCORE team is desired by the Ministry.
  

2.4 Employer’s organizations

• 
  ANDI is very happy with the SCORE project and it warmly supports the activities of SCORE. The textile and garment and security associations of ANDI have been actively involved in SCORE;
  
• 
  ANDI looks at SCORE as a productivity focused intervention with SME’s, but there is interest in CSR related aspects that are relevant for SME-sector to remain in business;
  
• 
  ANDI is interested to further market and promote SCORE in its sectoral chambers and among sector associations.
  

2.5 Trade Unions

• 
  The trade unions in Colombia (all three centrals) are very critical about the SCORE project and express that they have not been sufficiently consulted and involved in design and planning and implementation;
  
• 
  The trade unions (particularly CTC) are very concerned by the actions of SCORE in the Security Company sector. This sector is generally violating labour laws. One of the participating companies has misused the SCORE project for its own corporate communications and marketing. This is also affecting ILO’s good name in Colombia;
  
• 
  The trade union centrals indicate that their experience should be integrated in training of SCORE trainers and they can be involved in sector mapping and selection (and are currently doing so).
  
• 
  Possibly the participation of trade unions will increase now that a specific technical advisory committee can be established for the SCORE project.
  

2.6 Private sector support organizations

• 
  CTA and CNPML in Antioquia have been involved in promotion and marketing of SCORE modules in Antioquia, but the coordination and cooperation between SCORE and both organizations are not very active. Particularly CNPML expresses that there is no active cooperation and communication for quite some time. CTA is more positive and also still actively cooperating with SCORE, but also this organization indicates that cooperation could be much more active. CTA states that it has not heard clearly from ILO how it wants to continue with SCORE in the future. The vision around establishing a Centre of Excellence is not clear and as a result activities are at a low level at present;
  
• 
  CTA is sceptical about the sustainability perspectives of SCORE. The supply of training services is expensive and value for companies is not clear. Most BDS services in Colombia are provided for free (e.g. SENA);
  
• 
  The cooperation between the public BDS provider SENA and SCORE has not been prosperous. At the central level in SCORE phase one a large number of SENA trainers was trained in SCORE but due to the stopping of the cooperation between SENA and SCORE the SENA trainers are now non-active for more than two years and therefore have lost their certification;
  
• 
  SENA does not express an active interest to restart the cooperation, although they also formally don’t close the door;
  
• 
  The cooperation with regional SENA in Riohacha is better. In this region SENA trainers are being deployed in a Canadian funded SCORE-offspring project in Mining supply chains. In Riohacha the cooperation between SCORE and SENA is highly valued by local stakeholders.
  

2.7 Companies/Enterprises

• 
  The companies that were visited in this evaluation were highly satisfied with the SCORE training provided. Particularly the mix of classroom training and in-company support is highly appreciated;
  
• 
  The reports on capacity and willingness to pay for training are mixed. The two companies visited in this evaluation express interest and willingness to pay, but from most other stakeholder the evaluators hear that only subsidies SCORE provision is possible. Records of SCORE show that although cost-recovery in Colombia is still below 50% it is highest of all SCORE countries.
  

A small survey under SME’s that have benefited from SCORE shows the following appreciations on a five-point scale:

	Score
1. Relevance SCORE	5,00
2. Quality SCORE	4,33
3. Performance trainers	4,33
4. Effects on Management-Workers dialogue	4,67
5. Effects on productivity & profitability	4,33
6. Willingness to pay costs	4,00
7. Capacity to pay costs	4,00
8. Effects on worker’s organization & unionization	4,50
9. Effects on women participation	4,67

The table shows that SCORE is considered highly relevant by the enterprises and the effects on improved workers-management dialogue are notable, which is also confirmed by the enterprise walkthroughs. Somewhat surprising is the high score on effects on worker’s organization and unionization. Only in one of the enterprises visited there was a trade union. In the other companies there was limited awareness of trade unions role in labour relations and owners and managers mainly referred to the EIT’s as forms of worker organization.

3. 
   Main Findings
   

3.1 Relevance & strategic fit of the intervention

• 
  SCORE is relevant in the portfolio of ILO in Colombia and ILO’s presence in Colombia is important. There is no Decent Work Country Program for Colombia;
  
• 
  Colombia’s integration in regional and global markets after the civil war is progressing quickly. SME development is important to increase Colombia’s productive capacity and therefore SCORE is a very relevant project in this country;
  
• 
  SCORE is not sufficiently linked with the priority of improving labour relations in Colombia in the context of a highly hostile climate for trade unions. The SCORE project does not sufficiently address issues related to changing labour relations, except at the work floor level in concrete companies;
  
• 
  The sectoral choices of SCORE in Colombia have not been very fortunate. In the flower sector the supply of training is discontinued because there is no clear demand among companies in this sector and the sectoral association is not very interested in cooperating. Also the security sector that as added not as priority sector but as a result of a request of the ANDI/Association of Security Companies, was not a logical choice as this sector is mainly comprised by big (international) companies. Only the actions in the textile sector have been clearly more successful.
  

3.2 Validity of intervention design

• 
  In Colombia, ILO has not been able to establish sustainable relations with SCORE training services providers (SENA, CTA) and it gradually relied more on individual consultants contracted directly by ILO and managed from ILO. This is a weakness in intervention design, because the individual relations with trainers are time-consuming and expensive;
  
• 
  The set-up of SCORE in 5 modules proved to be difficult to market in Colombia. SCORE team is in favour in fine-tuning and tailoring SCORE modules to a specific module 1 and 5 combined product that can be offered for a lower price and where more possible external support (insurance companies) can be acquired.
  

3.3 Intervention progress and effectiveness

• 
  In spite of the slow start in 2013 and 2014, the project has regained speed and is now heading towards reaching its targets.
  

3.4 Efficiency of resource use

• 
  In retrospect, the not optimal cooperation between SCORE and SENA in phase I, has resulted in a considerable loss of investments of SCORE in training of trainers. The group of active trainers in Colombia is small and it is also largely dependent from individual consultants, which is laborious (and rather expensive) to manage.
  
• 
  The contracting and management of trainers is done by the SCORE team itself because there are no cooperation agreements with institutions that can provide such services. This leads to time-consuming micro-management by the small SCORE team and this limits its capacity to deal with other more strategic and externally oriented communication and coordination activities.