Ilo evaluation

This table shows that only a small percentage of enterprises provide inputs on enterprise improvements. They report a significant decrease of the number of accidents and grievances and a raise of the average wage of 6%. The estimated cost-savings are merely 3.12 US dollars, less than 0.1% of the savings reported in most other countries, which puts a question to the reliability of these data.

At the highest level, outcomes were reported on job-creation effects. The results (until 20 December 2015) are presented in the table below:

Enterprises Trained	139
# Enterprises for which data is available	129
% Enterprises providing basic reporting	93%
Job creation
Nr. of jobs created in enterprises trained	689
Nr. of jobs lost in enterprises trained	-558
Net change	131
Changes in employment status
Increase in casual workers	196
Decrease in casual workers	-109
Net change	87

Effects of SCORE on job-creation are significant, but it is also remarkable that part of the employment created is not in fixed jobs. Particularly casual jobs are created according to the ME& platform. This leaves a challenge to improve status of jobs in Colombia.

During the evaluation visit the evaluators have visited two SMEs to confirm achievement of outcomes in enterprise walkthroughs. The changes observed at SME level are presented in the table below. The average scores are given on a three-point scale (1=low, 2=sufficient, 3=good)

SME Walk Through Checklist	Score
1. Visibility CSR expressions	1
2. Visibility certificates (ISO etc.)	1,5
3. Overall cleanliness	3
4. Clear instructions workers safety	2,5
5. Visible OSH aspects	3
6. Signs energy efficiency	2
7. Gender specific provisions	2,5
8. Waste management systems	2,5
9. Facilities worker’s social wellbeing	2

Effects of SCORE are most visible in cleanliness of companies (results of 5-S) and in OSH measures introduced at the work floor.

Appreciation of workers on the application of SCORE in their companies are presented below on a five-point scale (1=not at all/very low, 2=low; 3=sufficient, 4=high, 5=very high):

Survey workers in SME’s benefiting from SCORE	Score
1. Awareness ILO/SCORE	4,17
2. Participation in SCORE	4,67
3. Quality SCORE	4,33
4. Performance trainers	4,17
5. Effects on Management-Workers dialogue	4,5
6. Changes in company after SCORE	4,67
7. Effects on labour conditions	4,5
8. Effects on worker’s organization & unionization	3,17
9. Effects on women participation	4,33

Workers are most positive about the effects of SCORE on their knowledge and experience and also on improvements in the workplace. Most critical are workers about the effects of SCORE to create institutional forms of worker’s organization in Indonesia.

2. Opinions and Appreciations of SCORE by partners and beneficiaries

2.1 SCORE team

The ILO SCORE team consists of three capable persons: the national project coordinator, a project officer and an admin secretary. Partners and contact persons at the companies that were visited value the team as professional, open in its communication and with a clear focus on teamwork and the collaboration with those who are involved. An important driver for the team members themselves is that through SCORE training and consultations behaviour and practices in SMEs have effectively changed and that these changes lead to more efficient production, better quality and an increase in productivity. Manoeuvring in the political landscape of Indonesia is sometimes difficult, but the team takes pride in its close cooperation with partners, especially the strong support it gets from the Directorate of Productivity and Entrepreneurship of MoM, not only at a national level but also in a growing number of provinces.
A small survey among SCORE staff shows appreciations on a five-point scale.

	SCORE
1. Integration SCORE in ILO policies & plans	4,00
2. Appreciation SCORE by Government (as perceived by SCORE officers)	4,50
3. Appreciation SCORE by Employers	4,00
4. Appreciation SCORE by Trade Unions	4,50
5. Appreciation technical support from central SCORE team by national teams	3,50
6. Perspectives future sustainability	4,00
7. Change behaviour SME’s after SCORE	5,00
8. Gender equity in SME's after SCORE	4,00

The survey responses are in line with the interview findings with the different stakeholder groups. SCORE has excellent relations with all stakeholders. The relation with trade unions in Indonesia is stronger than in most other countries.

2.2 SCORE trainer’s and SCORE training service organizations

A small survey among SCORE trainers shows the following appreciations on a five-point scale:

	Score
1. Growth SME demand for SCORE	4,67
2. Tailoring SCORE to SME needs	5,00
3. Appreciation SCORE by SME owners	4,67
4. Appreciation SCORE by SME workers	4,67
5. Quality SCORE modules & materials	4,67
6. Technical support from HEADQUARTERS SCORE team	5,00
7. Perspectives future sustainability	4,67
8. Change behaviour SME’s after SCORE	4,00
9. Gender equity in SME's after SCORE	4,33
10. Willingness & capacity SME’s to pay for SCORE	3,33

The trainers in Indonesia are very positive on the quality and effects of the SCORE project and effects. They are also happy with the technical support from ILO headquarters and believe that there is enough room for tailoring the training to the Indonesian context and specific needs of SMEs in the country.

2.3 Government

To strengthen the faltering economy and position in the Asian Community, raising the productivity and competiveness of SMEs are priorities of the Indonesian government. Especially the Directorate of Productivity and Entrepreneurship of MoM is very supportive towards the project. In 2012 the Directorate started to incorporate the SCORE methodology and modules into the basic curriculum for the training of its productivity instructors. It has since extended the implementation of SCORE trainings and is now reaching 10 provinces. Besides, the Directorate expressed the willingness to take on the role of SCORE National Centre.

2.4 Employer’s organizations

Because of the growing competition, especially in the Asian Community, APINDO finds SCORE an important project as it helps to increase quality standards, efficiency and productivity of SMEs. Insofar they directly benefit, enterprises will be willing to contribute to the costs. But it is important for SCORE to build its business case, according to APINDO, because economic hardship causes enterprises to think twice before they spend any money. APINDO does not seem to be the most committed partner in SCORE. One problem is maybe that its members are mainly larger companies and not SMEs. This makes the promotion of SCORE among members a bit difficult. APINDO would also like SCORE to focus more on the handicraft sector, but until now other members of the NTAC have rejected this proposal.

APINDO values the contribution of MoM, but at the same time feels that other ministries like the Ministry of Cooperatives and Small and Medium Enterprises, should be more involved in SCORE. Priorities of MoM are employment and labour conditions, whereas the Ministry of Cooperatives and Small and Medium Enterprises has it focus more on productivity and related issues. APINDO can play a role in establishing relations as it has high-level contacts at the various ministries. In addition, it can help to attract funding from larger companies that are a member and that work with SMEs in their supply chain. Until now the SCORE team was not able to use the full potential of APINDO being one of the members of the NTAC.
2.5 Trade Unions

The participation of trade unions in SCORE is limited primarily to serving on the NTAC. Trade unions consider SCORE as useful in different ways. At factory level the project improves the collaboration between workers and managers and helps to increase efficiency and productivity, which in the long run will lead to higher wages. Through SCORE workers are empowered and they become more self-confident because of their participation in SCORE modules. SCORE adds value, according to trade unions, in that it not only provides training and theory, but also results in an EIT that formulates an improvement plan with regular meetings with management and that leads to productivity gains and better social dialogue at the work floor. On a more general level the SCORE project helps to improve the collaboration between employers, trade unions and government. SCORE enables trade unions to demonstrate that they are not only opponents to employers but are also willing to help employers to raise the competiveness of their enterprises. As they explained during the interview, this will benefit the workers as well.

2.6 Private sector support organizations

During the evaluation visit a meeting was planned with SOI. We did not interview the other service provider BEDO. SOI values the technical support and guidance received from the ILO SCORE team. It feels that there is a demand of SMEs for SCORE trainings and that the project is well appreciated by most SME owners, managers and workers. Like APINDO it believes that based on the positive results, SMEs, especially the bigger ones, will be willing to pay part of the costs. One of the main bottlenecks is the participation of the management. Sometimes companies send lower managers to the training. Higher management is not involved in the training or in the EIT. Because of a lack of support and decision-making power, the impact of SCORE in such cases is mostly limited. To get sufficient commitment from higher management and owners, SCORE needs to demonstrate its results. In addition, SOI believes that more attention could be given to needs assessments to be able to target the training and consultations more on the specific needs of companies or certain sectors.

2.7 Companies/Enterprises

More than 800 managers and workers of 139 enterprises have attended classroom training for the various SCORE modules. Regarding their satisfaction with the training, 77% of the participants rated the training as good or very good (four or five on a one to five scale). The percentage of SMEs enrolling in at least one additional module after the mandatory module 1 is 24%, which is lower than in most other SCORE countries. The most notable effect, according to the managers and workers that were interviewed during the evaluation visit, is that workers and managers cooperation has improved and that workers are more empowered in their companies. SCORE leads to a change in mind-set, a sense of ownership and a more open dialogue at the factory level. All interviewees are certain that this will enable their company to increase its productivity and competitiveness by making products better, faster and cheaper.

A small survey under SME’s that have benefited from SCORE shows the following appreciations on a five-point scale:

	SCORE
1. Relevance SCORE	5,00
2. Quality SCORE	4,80
3. Performance trainers	5,00
4. Effects on Management-Workers dialogue	4,80
5. Effects on productivity & profitability	4,60
6. Willingness to pay costs	3,60
7. Capacity to pay costs	3,80
8. Effects on worker’s organization & unionization	4,00
9. Effects on women participation	4,80

The table shows that SCORE is considered highly relevant by the enterprises and the effects on improved workers-management dialogue are notable, which is also confirmed by the enterprise walkthroughs. The scores on the capacity and willingness to pay part of the costs of SCORE trainings are lower.

3. 
   Main Findings
   

3.1 Relevance & strategic fit of the intervention

• 
  Due to their economic contribution to the overall economy and their role as major source of employment in the country, SMEs are very important in Indonesia. However, the potential contribution that SMEs can make, both socially and economically, is much greater than is being achieved at present. Poor workplace practices and a lack of communication between workers and management are a significant part of the problem. That is why all stakeholders agree that the SCORE project is important in that it can help enterprises to find a better road to productivity and growth that benefits both workers and employers.
  
• 
  Although needs assessments can be improved, all stakeholders agree that the SCORE modules and consultations are well tailored to the needs and demands of SMEs.
  

3.2 Validity of intervention design

• 
  The project shows good results at the company level. Additional to results in quality management, better organized and cleaner factories, storage, energy use and efficiency, most notable and relevant in the light of the ILO mandate according to the NTAC members is that workers and managers cooperation has improved and that workers are more empowered in their companies.
  
• 
  An important added value of SCORE to existing projects is that it combines theory (training) with practice. It is a structured method in which training is followed by the installation of Enterprise Improvement Teams, agreed action plans between workers and management, implementation schedules, regular meetings, tangible improvements and cooperation. In addition to the trainings, consultants visit companies to help with the implementation of improvement plans. The SCORE project combines training and theory with a hands-on approach.
  
• 
  SCORE Indonesia has taken steps to improve the degree of mainstreaming and integration of gender aspects into the project. All 5 modules are gender-mainstreamed and all ILO SCORE invitations to SMEs to participate in any kind of SCORE training, always has a statement about gender equality. Since the beginning of Phase II, the number of female owned enterprises has gone up from 12 to 25 enterprises. The number of female trainers has increased from 8 to 27, and the number of female staff trained in workshops went up from 150 to 309 persons.
  

3.3 Intervention progress and effectiveness

• 
  So far SCORE Indonesia achieved a number but not all of its objectives. Two private service providers have embedded SCORE training in their SME service portfolio. So has the training centre of the Directorate of Productivity and Entrepreneurship of MoM, bringing the total number to 3 institutions that are working with SCORE trainings (target 2). Together with its partners the ILO SCORE team is now running trainings in 11 of the 34 provinces of Indonesia. There are 63 active SCORE trainers, of which 27 (43%) are women. Because the Directorate of Productivity and Entrepreneurship, as future National Centre, is working on a certification standard for SCORE trainers in Indonesia, none of the trainers have been certified until now. The number of SMEs trained is 139 (target 153), of which 20% is female owned (target 15%) and 37% of the participants in SCORE training were women. The number of policy makers, labour inspectors and social partners trained, is slightly below target. The percentage of SMEs taking up more than 1 module is 24% (target 30%). The ILO SCORE team is making an extra effort to raise this percentage. Because of this and due to delays in decision-making and funding by MoM, a limited number of 5 new enterprises have been trained in 2015.
  
• 
  A key achievement of SCORE Indonesia is the strong involvement of the Directorate of Productivity and Entrepreneurship of MoM. Although no agreement has been signed yet, the Directorate has expressed its willingness to take on the role of SCORE National Centre in Indonesia.
  
• 
  Because of a change in personnel and delays in the funding by MoM, the cost recovery rate has dropped in 2015, but the SCORE team is convinced that this will only be of temporary nature. Service provider BEDO, who fairly recent joined SCORE, has been successful in attracting external funding for SCORE activities.