Ilo evaluation

Focus Area of Evaluation: Intervention progress and effectiveness	Is the project on track in delivering its outputs in all countries?	Extent to which project activities are implemented according to timeframes Availability of documentary evidence of implementation	Tri-partite stakeholders Service Providers Business owners Workers SCORE monthly, biannual & annual reports	Interviews (ind. & group) Focus groups Survey Desk-study SCORE Data-base and M&E system
	To what extent has the project so far achieved its objectives (incl. the cost recovery plan) and reached its target groups? Do project outcomes contribute to gender equality?	Number of institutions that have embedded SCORE training in their service portfolio % of training costs recovered by non-ILO resources so far Number of institutions and corporations that sponsor/pay for SCORE % of Modules delivered with medium or high independence Number of SMEs trained and visited (% of female owned SMEs) and % of SMEs that have formulated and implemented action plans(s) after training Number of male and female managers and employees trained and their perception and appreciation of quality of trainings % SMEs reporting cost saving, reduction absenteeism and employee turnover (% women), reduction of defects, waste, accidents, improved energy efficiency and increased employee satisfaction, both male and female Number of new partner initiatives promoting responsible workplace practices after SCORE training	Tri-partite stakeholders Service Providers Business owners Workers SCORE monthly, biannual & annual reports	Interviews (ind. & group) Focus groups Survey Desk-study Gender assessment SCORE Data-base and M&E system
	Concerning the institutional level, how far has the capacity of partner organizations been built in relation to delivery of the outputs/objectives?	Number of representatives of industry associations and training institutions trained (% of women) Number and quality of business plans incl. marketing & communication plans for SCORE training developed by partner organizations Presence of a national centre of excellence for SCORE Number of national expert trainers available per country Existence of Trainer of Trainers curriculum and number of trainers certified in SCORE (% of women) Availability of training materials translated and adapted to national context Number of partners trained to include gender equality concerns and extent to which gender is mainstreamed in training modules Functioning of M&E database	Tri-partite stakeholders Service Providers Bi-annual & annual reports	Interviews (ind. & group) Focus groups Survey Desk-study Gender assessment
	Concerning the enterprise-level, is the training programme effective in leading to the desired impact on enterprises and their workforce? Are the factory improvements lasting? Are SMEs enrolling in more than one Module? If not, why? Has the project targeted the right market and created the right channels for the SMEs to enrol in more than one Module?	Number of SMEs trained in Module 1 % of SMEs taking up more than one training module Appreciation of the quality, relevance and extent to which trainings address companies’ needs % of SMEs that have formulated and implemented action plans(s) after training % SMEs reporting over a longer period of time cost savings, reduction absenteeism and employee turnover (% women), reduction of defects, waste, accidents, improved energy efficiency and increased employee satisfaction (male and female) Argumentation of SMEs that did not follow more than one module	Service Providers Business owners Workers SCORE monthly, biannual & annual reports	Interviews (ind. & group) Focus groups Survey Desk-study Company walk-through Gender assessment SCORE Data-base and M&E system
	What obstacles did the project encounter in project implementation? What corrective action does the project need to take to achieve its objectives?	This question does not refer to indicators and will be answered in the chapters of conclusions and recommendations.	Tri-partite stakeholders Service Providers Business owners Workers SCORE monthly, biannual & annual reports	Interviews (ind. & group) Focus groups Desk-study
Efficiency of resource use	Does the project make efficient use of its financial and human resources?	Comparison of total staff time dedicated to SCORE at country and global level Turnover over staff Amount and frequency of over- and/or under-expenditure in project / per country	SCORE and implementing org staff Subcontracted service providers SCORE & implementing org staff Subcontracted service providers SCORE monthly, biannual & annual reports	Interviews Desk Study
	Is the implementation strategy cost-effective?	% Administration costs Comparison costs per module/trainer/trainee per country	SCORE and implementing org staff Subcontracted service providers SCORE monthly, biannual & annual reports	Interviews Desk Study
	Is the distribution of resources between staff and activities optimal?	% Administration costs Distribution budget global and country level	SCORE monthly, biannual & annual reports	Interviews Desk Study
	Were the intervention resources used in an efficient way to address gender equality in the implementation?	% of budget invested in specific gender related activities Project expenditures per female beneficiary compared with male beneficiaries	SCORE and implementing org staff Subcontracted service providers SCORE monthly, biannual & annual reports	Interviews Desk Study Gender assessment
Focus Area of Evaluation: Effectiveness of management arrangements	Are time frames and work plans respected? How are contingencies dealt with? To what extent corrective action is taken when required?	Amount and magnitude of delays compared per country Amount and kind of external factors influencing implementation. % of cases in which proven mitigating actions were taken	SCORE and implementing org staff Subcontracted service providers SCORE monthly, biannual & annual reports	Interviews Survey Desk Study
	Is the management structure effective?	Appreciation of ILO staff, implementing partners and sub-contracted service providers on the effectiveness of management structure	SCORE and implementing org staff Subcontracted service providers	Interviews
	Has the project staff sought and received adequate support from the global component, relevant ILO units and offices, including gender expertise when needed? Does the country project staff have sufficient authority/delegation in executing and managing the project (e.g. technically, financial and admin. management)?	Appreciation of ILO staff, implementing partners and sub-contracted service providers on the degree of autonomy and effective and efficient decentralised operations at the country level Appreciation of ILO staff, implementing partners and sub-contracted service providers on the timeliness and quality of central service delivery, including specific gender expertise	SCORE and implementing org staff Subcontracted service providers	Interviews
	Should the project develop regional hubs for support?	This question does not refer to indicators and will be answered in the chapters of conclusions and recommendations.
	Are donor relations, administration of processes, and project monitoring by donor Headquarters and country staff effective?	Appreciation of ILO staff of administrative dialogue and cooperation and specific requirements of SECO and NORAD Frequency and kind of contact/dialogue between SCORE and national representative offices of SECO and NORAD at country level	SECO & NORAD ILO central and decentralised staff	Interviews
	Are the Global and National Tripartite Advisory Committees functioning and what value do they add?	Frequency of meetings and attendance by the tri-partite members Appreciation of dialogue and outcomes of N and/or G TAG meetings by tri-partite partners.	SECO & NORAD ILO central and decentralised staff ITUC and IOE (and their national members) Minutes G&N TAC	Interviews Desk Study
	Is project systematically and appropriately monitoring, docu-menting & communicating results, including on gender, at the country and global level? Is M&E system practical, useful & cost effective for project management? Does manage-ment use information generated by M&E system to assess progress against the objectives (including gender-related results) and take necessary adaptive measures when required? Should indicators be revised?	DCED qualification of SCORE M&E systems Appreciation of ILO staff and implementing partners of quality and usefulness of information Number and kind of revisions and adaptations that were made based on interpretation of M&E data, including specific gender actions Amount of time and effort needed to generate M&E data and to retrieve M&E data from on-line database and SMART sheets % indicators that is not measured and/or are systematically not achieved	DCED audit report M&E systems Monthly, bi-annual and annual reports SCORE staff and implementing organizations staff	Interviews Desk Study M&E system assessment Gender assessment
	How effective is the project in sharing good practices between country components and communicating success stories and disseminating knowledge internally and externally (including gender-related results and knowledge)?	Number and kind of information exchange and meetings at the ILO, regional, global level Number and kind of information exchange and meetings with external stakeholders at regional, global level Incorporation or replication of SCORE lessons in other countries and or by other stakeholders (gender lessons specified)	SCORE and implementing organizations staff External stakeholders Annual Reports	Interviews Desk Study
Focus Area of Evaluation: Impact orientation and sustainability of the intervention	How effectively has the project built national ownership and capacity of people and institutions? Are national partners (employer organizations and industry associations) willing and able to continue the project after funding ends?	Appreciation of SME owners of SCORE programme Self assessment of capacity of training service providing organizations Degree of willingness to pay for training services Appreciation of employer organizations and business associations of the programme Willingness to continue the project after ILO moves out	SME owners Training providers Tri-partite partners SCORE and implementing organizations staff	Interviews Focus group meetings Survey
	Are there business models applied in the different countries that seem more promising to reaching financial sustainability?	Degree of financial sustainability of other enterprise training and capacity development programmes (as far as information can be found on this) Number and kind of proposals of business community for other business models for enterprise training	SME owners Training providers Tri-partite partners SCORE and implementing organizations staff	Interviews Focus group meetings
	Are the gender-related outcomes likely to be sustainable?	Appreciation and assessment of employers and workers on quality and sustainability of gender outcome Evaluators’ gender assessment of outcomes	SME owners Training providers Tri-partite partners SCORE and implementing organizations staff	Interviews Focus group meetings Gender Assessment
	Has the project reached sufficient scale and depth to justify the investment?	Appreciation of tri-partite stakeholders on scope and scale of SCORE project Historical growth and outreach of SCORE among service providers, trainers and companies Evaluator’s assessment	SME owners Training providers Tri-partite partners SCORE and implementing organizations staff	Interviews Focus group meetings
	Has the project found the right balance between scale and depth and the trade-off between them?	Appreciation of tri-partite stakeholders on scope and scale of SCORE project Evaluator’s assessment on Balance	SME owners Training providers Tri-partite partners SCORE and implementing organizations staff	Interviews Focus group meetings
	Are the approach and its results likely to be up-scaled or replicated?	This question does not refer to indicators and will be answered in the chapters of conclusions and recommendations
	Is the project a cost-effective way to improve productivity and working conditions in SMEs?	This question does not refer to indicators and will be answered in the chapters of conclusions and recommendations

ANNEX 13 Terms of Reference

SCORE – Sustaining Competitive and Responsible Enterprises

Independent Mid-Term Evaluation

Title:	Sustaining Competitive and Responsible Enterprises (SCORE)
TC Code:	Global SCORE (GLO/13/55/MUL) SCORE China (CPR/13/50/MUL) SCORE India (IND/13/50/MUL) SCORE Indonesia (INS/13/51/MUL) SCORE South Africa (SAF/13/50/MUL) SCORE Ghana (GHA/13/50/MUL) SCORE Colombia (COL/13/50/MUL) SCORE Viet Nam (VIE/13/51/MUL)
Backstopping unit	SME Unit
Type of evaluation	Independent Mid-Term Evaluation
Evaluation Manager	Annie van Klaveren

List of Acronyms

AA	Administrative assistant
BDS	Business development services
CTA	Chief technical advisor
DCED	Donor Committee for Enterprise Development
DWCP	Decent Work Country Programme
EM	Evaluation manager
EVAL	Evaluation office
HEADQUARTERS	Headquarters
ILO	International Labour Organization
IOE	International Organization of Employers
ITUC	International Trade Union Confederation
KPIs	Key performance indicators
M&E	Monitoring and evaluation
NORAD	Norwegian Agency for Development Cooperation
NPC	National project coordinator
NPO	National project officer
P&B	Programme and budget
PPPs	Public private partnerships
SCORE	Sustaining Competitive and Responsible Enterprises
SECO	Swiss State Secretariat for Economic Affairs
SMEs	Small and Medium Enterprises
TA	Technical assistant
TAC	Tripartite Advisory Committee
TC	Technical committee
TO	Technical officer
UN	United Nations
UNDAF	United Nations Development Assistance Framework
UNEG	United Nations Evaluation Group
USD	United States dollar