Ilo evaluation

3.5On M&E and reporting

The M&E system underwent an audit against the DCED results measurement standard and passed with good results. The M&E system of SCORE is built on the following blocks:

• 
  M&E digital platform - web-based tool to capture new developments in SCORE on a daily basis. The M&E platform generates basic country reports and a global report on SCORE that is entirely up to date (the evaluators have used the M&E data of 20 December 2015).
  
• 
  Smart sheets – excel sheets that can be filled out on-line by country teams and trainers. Two sheets are used for M&E: a) country work-plans, with a traffic-light system to capture progress in implementation, and b) performance sheets that compare performance against targets.
  
• 
  A third (offline) excel sheet was developed to visually present the intervention logic of SCORE at country level. These sheets were developed for all SCORE countries but are now only still in use in India.
  
• 
  Quarterly reports prepared and submitted on the progress per country by the SCORE teams.
  
• 
  At global level, the central SCORE team prepares bi-annual progress reports.
  
• 
  In addition to these blocks there is a separate website (papyrus.ilo.org) that functions as a recourse base for SCORE teams and trainers. On this website a lot of information on SCORE can be found, including training manuals.
  

Key Performance Indicators

The Key Performance Indicators (KPIs) form the core for measuring outcomes at the enterprise level. All building blocks compile information on these KPIs. The whole set of KPIs for SCORE is presented below.

table : Key Performance Indicators used in the SCORE project

KPIs (average per enterprise): # EIT meetings per month # Improvement plans completed # Gender-sensitive practices # Employee suggestions % Change in grievances received % Change in defect rates % Change in energy usage per production unit % Change in absenteeism % Change in worker’s turnover % Change in accidents logged % Change in wage per employee Estimated cost-savings (in local currency)	Module 1: Existence of daily worker-management meetings Women & men in EIT Existence of employee suggestion schemes Application of 5-S Cost savings due to SCORE training
Module 2: Existence of quality policy Systematic analysis of causes defects Quality info shared with employees Reduction in defects	Module 3: Reduction of use materials Reduction of waste Existence of machine maintenance system Reduction of energy consumption
Module 4: Existence of human resource policy Existence of job descriptions Increases in wages Reduction in worker’s turnover	Module 5: Existence of worker-management occupational health and safety committee Existence of health and safety policy Reduction in workers’ complaints Reduction of accidents Reduction of absenteeism

Source: SCORE M&E platform

The general KPIs are quantitative and are aggregated averages for all companies in a specific SCORE country. Also at the specific module level KPIs are measured, but this is done in a bi-nominal way. The SMEs that change their response to a specific indicator from no to yes (false or true) are counted. Where possible and relevant, the KPIs also present gender-disaggregated data.

While the M&E system of SCORE seems rather sophisticated, in practice mostly the output and activity based data are reliable and used for reporting on SCORE. A considerable number of SMEs provides outcome-monitoring information. This is illustrated in the figure on the next page.

figure : Percentage of SMEs reporting on changes in key performance indicators

Source: SCORE M&E platform 20-12-2015
On average, 75% of the companies provide some information on outcome indicators. In South Africa, Bolivia and China this percentage is considerably lower, while the percentage in Peru, Indonesia, Colombia and Vietnam the percentage is somewhat higher. The percentages in the figure above don’t reflect to which extent such reporting has improved over time, but in several countries, the SCORE teams indicated that over time SME reporting has improved considerably. But in spite of improvements, the M&E platform and SCORE annual reports also show that even while more companies report on basic outcome indicators, a much more limited number of companies (and consultants) systematically reports on changes under several or all KPI’s. A quick scan of enterprise reports shows that reports often contain only information on a few indicators and not the full package.

Limitations of M&E reporting

During the interviews, the evaluators received feedback about technical problems while uploading data, such as large amounts in currencies, but also the fact that sometime the same changes in different situations are reported positively and negatively (a decrease is not always reported as a negative figure. These technical difficulties and unclarities create more room for errors in reporting. Overall, there is also quite some internal dissatisfaction with the inability of the M&E system to allow queries of outcome monitoring information and make this information available for analysis. It is time-consuming to enter data into the system and only a limited amount of data for analytical use comes out of it. Not all KPIs are considered relevant and some of them seem to exist more to generate centralized reporting to donors than to help to guide, steer and fine-tune the country-level implementation, let alone learning. The obvious result of these limitations is that the SCORE quarterly and bi-annual reports merely provide information at the activity and output level and at this level reports can be considered of good quality. Data generated on outcomes, however, do not generate sufficient quality data to enable a good outcome level monitoring of the programme at the enterprise level. This is in line with the findings of the DCED audit.

Flaws in outcome reporting

Until recently, bi-annual reports on SCORE only presented data at the output level and no systematic information was given on the outcomes. Starting in 2015, ILO has started to aggregate once a year the outcome figures and these for the first time were included in the first report of 2015. However, reporting on outcome is more problematic than on outputs. The analysis of monitoring data in this evaluation shows that M&E data on outcomes in SCORE are not seldom erratic, incomplete and almost impossible to interpret. At the enterprise level, indicators used for outcome monitoring in the M&E platform are the following.

table : Enterprise performance indicators used in SCORE monitoring

Average # EIT meetings/month

Average # EIP projects completed

Average # gender-sensitive practices

Average # employee suggestions/month

Average % change # grievances

Average % change defect rate

Average % change energy usage

Average % change absenteeism

Average % change labour turnover

Average % change # accidents logged

Average % change average wage/employee

Estimated average cost-savings due to SCORE (in local currency)

Source: ILO SCORE M&E platform

Taking the last indicator as an example, in table 6 the evaluators have recalculated in US dollars the estimated average cost-savings for SMEs due to SCORE in each country.

table : Estimated costs savings per enterprise as result of SCORE modules

	China	Colombia	Ghana	India	Indonesia	Peru	Vietnam
Estimated average cost-savings (USD)	22.769,00	359,00	1.189,00	939,00	3,12	2.969,00	0,15

Source: ILO SCORE M&E platform (20-12-2015). Savings in local currency converted in USD (20-12-2015).

In South Africa and Bolivia no information is collected on cost savings. The table above demonstrates that savings range between 0.15 US dollar in Vietnam to almost 22,000 US dollars in China. This shows that the information collected on the outcome indicators is not always reliable and cannot be compared. The SCORE team at ILO headquarters also came to this conclusion that it is very difficult to estimate cost-savings and stopped analyzing these data and reporting on them, although these data are still collected in the M&E platform.

At the level of the developmental impact of SCORE the following indicators are used.
table : Indicators for job creation and employment status

Job creation

Nr. of jobs created in enterprises trained

Nr. of jobs lost in enterprises trained

Changes in employment status

Increase in casual workers

Decrease in casual workers

Source: ILO SCORE M&E platform

The M&E data on job creation cover a range from 18 jobs in South Africa to 1,595 new jobs in Vietnam for the entire duration of the SCORE programme. In China, 707 job-losses are reported. Negative net employment effects are also reported in India, Peru and South Africa. These data are somewhat difficult to interpret as the number of jobs created or lost depend on many factors and might also have been assessed in different ways. Additionally, the evaluators have shown that there very well can be positive effects on workers and quality of jobs, while there are negative employment effects. Therefore job creation and employment status data should be used with caution and not be considered as direct performance indicators for SCORE. The SCORE team at ILO headquarters stopped using these indicators to monitor the developmental impact, because it does not have the means in most cases to attribute changes at this level to SCORE training. M&E data on other outcomes for the specific countries are very different and cannot be compared, let alone aggregated. Annex 11 contains a detailed analysis of the monitoring information generated by the M&E platform. From this analysis it becomes clear that the M&E system does not generate sufficient information on the outcomes of SCORE.

Level of aggregation

In interviews during the country visits SCORE staff, trainers and SMEs reported the following causes for the flaws in the outcome reporting.

• 
  Different trainers report differently on effects of training.
  
• 
  Each sector is different. In service sectors other changes are achieved than in manufacturing, while for example some sectors in the manufacturing industry also have a higher potential of energy savings than others. Whether energy savings are higher in one SCORE country than in an other, is probably related to the choice of the sector and not to the performance of SCORE in that particular country.
  
• 
  All SMEs are unique in size and situations and therefore potentials for savings and job-creation are different. While aggregating different data, this variety gets lost and what remains are average numbers that do not carry a lot of informational value.
  
• 
  The source of reporting (usually SMEs themselves) is biased and not always reliable. They may want to exaggerate effects or do the contrary, depending on their situation and interests.
  
• 
  Indicators are often not relevant for the SMEs and not very well understood.
  

Indicators like savings in energy usage and number of accidents are useful at the level of individual enterprises and maybe also at a sectoral level. But these indicators do not make much sense at a higher aggregation level, as sectors are different and difficult to compare. SCORE should rethink the indicators that are used for outcome reporting. In all countries, the SCORE teams indicate that the M&E data that are currently generated at the outcome level make little sense and are quite cumbersome to collect. There is a general interest of SCORE implementers and stakeholders to apply changes in the M&E system. Because of the weaknesses in the M&E system, one needs to look at specific examples to analyze the effects of SCORE training on SMEs. Case studies and testimonials of SME owners, managers and workers are collected in most SCORE countries and these examples show a high impact at the company level. However, this only provides anecdotic proof.