Introduction [TJ]

DESCRIPTION OF THE STUDY

3.1 Scope and Objectives
This study specifically evaluated the school networking projects, otherwise known as SchoolNets, to which Acacia provided support in varying degrees during the period 1997 to 2000. It signals the end of the first phase of Acacia’s support for various school networking projects. The projects included in the evaluation were selected from the following countries: Angola, Lesotho, Mozambique, Namibia, Senegal, South Africa, Uganda, Zambia and Zimbabwe.
The basic unit of analysis for the study was the specific SchoolNet project, with particular reference to the component supported by Acacia. Its main areas of investigation were on the nature of the inputs, activities, outputs, effects and lessons of these SchoolNets. Given the recent nature of these SchoolNet developments, it is still too early to conduct a comprehensive impact assessment on learning and teaching. Such a study may be conducted at a later stage.

3.1.1 General Objectives

The general objectives of this study were to:

• 
  Document school networking projects supported by the IDRC in Africa between January 1997 and July 2000;
  
• 
  Inform subsequent phases of evaluation on school networking;
  
• 
  Inform the IDRC’s future programming in ICT application in education;
  
• 
  Inform subsequent evaluation processes and projects;
  
• 
  Take the first step in the establishment of a repository of information regarding school networking experiences in Africa that can be shared with a broad range of interest groups and stakeholders; and
  
• 
  Inform potential pan-African research studies related to the social dimension of ICT application in Africa.
  

3.1.2 Specific Objectives

The specific objectives of this study include the following:

• 
  To investigate various start-up models for school networking in an African context;
  
• 
  To ascertain the experience of more mature SchoolNets with particular reference to the following:
  
• 
  Establishing computer access and connectivity to the Internet;
  
• 
  Developing the capability of educators to utilize ICTs to enhance learning and teaching;
  
• 
  Developing local education content; and
  
• 
  Ascertaining the extent to which national policy on the use of ICTs in education has been developed either under the influence of the SchoolNet or whether it has influenced the development of a SchoolNet.
  

3.2 Evaluation Approach and Methodology
This evaluation study was conducted within the framework laid down by the Acacia Evaluation and Learning System (ELSA). ELSA makes a distinction between ongoing performance monitoring, which tracks whether actual performance and results are on target, and more discrete evaluations, discontinuous data collections, or analytical studies which assess issues such as effectiveness, sustainability and the impact of programs. ELSA is based on the premise that a series of feedback mechanisms are in place between the various stakeholders so that they will receive timely information to support management, investment or other decisions.¹⁹³
In the case of school networking projects up to July 2000, the projects were evaluated retrospectively. These summative evaluations are intended to serve as baseline data for continuous, formative evaluation in subsequent phases of Acacia’s school networking projects (July 2000 onwards), should these projects continue.
Figure 1.1 illustrates the process that was followed in each of the SchoolNet projects under evaluation: What it illustrates is the level of interaction between the in-country evaluation teams and others involved in the evaluation process.



Figure 1.1 Process Followed for the Evaluation of Acacia-Supported

SchoolNet Projects in Africa
Following preliminary discussions, a research design workshop was held in July 2000. This was

attended by SchoolNet project managers from Mozambique, South Africa and Zambia, and representatives from the IDRC’s Acacia program. The workshop agreed to conduct in-depth case studies on the SchoolNet projects in the four Acacia national strategy countries: South Africa, Mozambique, Uganda and Senegal.

Existing documentation was reviewed and research instruments were designed and piloted. Appendix 1 includes an example of an evaluation framework developed for the SchoolNet South Africa case study and which formed the basis for the development of the research instruments.

With subsequent refinement of the instruments, fieldwork was undertaken which entailed site visits, and data collection through questionnaires and interviews. A feedback workshop was held in January 2001 with the evaluation team researchers. Representatives from the school networking agencies in each of the nine pan-African research study countries were present to comment on the research that had been undertaken in their specific contexts. A final research report was then made available with the intention of producing it in multiple formats (book format, Website and CD-ROM versions).

3.3 Research Questions
The primary evaluation questions for this study were:

I
n addition to these primary research questions, the following themes were explored: context; infrastructure and connectivity; content development and teacher training; policy and project planning; project activities and implementation; effects of the project and the lessons learnt. There were however variations in focus between the country studies, as circumstances demanded. A detailed list of the types of questions posed is included in Appendix 2.

3.4 Data Gathering Techniques
The following data gathering techniques were applied:

• 
  Document reviews
  

This included inter alia project proposals, progress reports, planning documents, correspondence, annual reports and Websites as well as policy documentation.

• 
  In-depth interviews (face-to-face and telephone)
  

Key stakeholders included project staff and committee members, government officials, teachers and students, Acacia staff and any other key beneficiaries identified by the evaluation team.

• 
  E-mail surveys
  

As a data collection technique, the e-mail survey proved useful because a number of countries were covered at much lower direct research costs, and respondents had time to consult other relevant documents and people, when answering the questions. However, the limitations with this technique were the lack of control by the research team over the process of completing the questionnaire, delays resulting from technical problems, and misinterpretation of questions.

• 
  Site visits (and observation)
  

Details are provided for each of the case studies presented in this study.

• 
  Photo-documentation, where available.
  

Validation of the research results included:

• 
  Interviewing different stakeholders on the same issues in order to ascertain a range of views and opinions;
  
• 
  Conducting site visits and testing observations against the points raised in interviews;
  
• 
  In some of the country studies, researchers tested the computers by trying to log on, access Internet, send e-mail and open other programmes such as MS Word, Excel and PowerPoint. The latest documents on the computers were also checked to find out if the computers had actually been used; and
  
• 
  A feedback workshop was organized with SchoolNet project managers in January 2001. Here the research findings were presented to the SchoolNet project managers, evaluators and donor representatives who gave extensive comments on the findings. This workshop included a site visit by the entire workshop team to a local school in South Africa and tested the findings of the visit against the research findings on the SchoolNet study.
  

3.5 Challenges in the Research Process
A range of factors influenced the outcomes of the research particularly during the data collection phase. These included:

• 
  The timing of data collection. The data was collected at the time when students were writing exams, which limited the number of teacher and student time. As a result, the number of teachers and students that could be interviewed was very limited.
  

• 
  The informants from different African countries spoke English as a second language and English was the language of communication during the interviews. This affected the quality of the information provided by the informants.
  

• 
  Non-availability of other key respondents. In some countries, the key respondents were not available. In Angola for example, the researcher only interviewed one respondent. This has major implications for the extent to which one can generalize the findings of this evaluation study.
  

• 
  A related factor is that some SchoolNet projects had no people employed on a full-time basis. The board and steering committee members were employed elsewhere and their participation in the SchoolNet project was on a voluntary basis, making them inaccessible for an interview. Two cases in point were SchoolNet Zambia and SchoolNet Namibia, where the researcher had to spend more days than initially planned for the site visits because some of the crucial members of the board were not available.
  

• 
  A number of the SchoolNet projects had multiple sources of funding and in some cases such as SchoolNet Zimbabwe, Acacia support was minimal. This affected the assessment of the direct impact of Acacia’s support in particular.
  

• 
  Some of the projects did not have a management structure. Lesotho and Zimbabwe are cases in point. This made it difficult, particularly in the case of the start-up projects, to access key players.
  

• 
  Generally, there were poor responses to the e-mail surveys, where these were undertaken.
  

• 
  The SchoolNet projects were at different levels of development. Some projects such as the one in Lesotho had only just started their activities during the time of data collection, which made it difficult for some questions to be probed.