The e-Tools (1) Report: Pedagogic, Assessment and Tutoring Tools

Contextualisation by the Author¹

It was easier for me than for some other authors to revisit this topic of my report, because in a sense I never escaped from it – but maybe that makes it harder to come back with a fresh eye. I set up and ran this study on procurement having been involved with IT procurements at universities and in JISC circles for some years previously; and after the study, I continued involvement with the topic on further HEFCE contracts and indeed while at UKeU, where analysis of similar (and possibly competitive) systems and providers was one of my key tasks.

But there were long periods in my post-report career where procurement of LMS components was not my prime focus. So what do I find when I turn back to the report with a fresh-ish eye?

My headline conclusion is that a “proper procurement process” is still essential. There might have been a couple of years between “then” and “now” when studies of the sort that are revealed in this chapter would have been seen as a waste of time. After all (many said in UK HE), there were essentially only two suppliers; they both offered “soup-to-nuts” functionality of a similar nature and so it all came down to price, and any special deals one’s university could negotiate. But this was a higher education view (the situation was always much more multi-polar in further education); and in reality it was never quite thus in HE since the specialist suppliers (like Questionmark) did not go away – indeed, they thrived.

Even if there were an element of truth in the above, the game has changed, due to interoperability and open source. (Although, I have to note, with some pride, that the phrase “component-based architecture” was used even in the original Sheffield Hallam University bid for this study – see appendix F.) “Suddenly” (it seems to some) there is again a plethora of suppliers. Some cover the board, while others are increasingly specialist; yet others from the open-source community (communities, really – there are many subspecies) hardly seem like suppliers at all. But interoperability, within a component-based framework, should make it all happen, all “join up”. In time.

On learning-technology standards and interoperability, despite progress not being as fast as the devotees wanted (among whom were some of my staff); there has been steady progress, and it is now inconceivable to think of an e-learning world without learning-technology standards. Yes, perhaps too much of this is still dominated by US and US-derived agencies; yet countries like the UK (and the Netherlands, among others) who want to have a voice at the top table can and do achieve that. So at least the unpleasant Europe-USA standards rivalry of the early part of this millennium is now part of history, where it should stay. Indeed, the UK generally has nothing to be ashamed of, with a number of world-class standards experts from the UK – in CETIS, at vendors and as consultants – including a surprising number from my own town of Sheffield (partial but not total explanations focus on the University for Industry, or Ufi, and FD Learning, but there is more to it, and them, than that). On interoperability, it remains too hard, but it is slowly getting easier, and now we all know and accept (vendors too) that we have to do it.

One new thing (to me) that I did discover from contextualising this report is that although there have been many changes and much consolidation in the LMS industry, remarkably few vendors have gone out of business without leaving an “inheritor” organisation. It also became obvious that many suppliers not seen recently in UK HE were “not dead but sleeping” – actually, they were and are wide awake, pursuing what they judge as more lucrative opportunities in other countries or sectors. But it is a borderless world and the border between education and training, which this report foresaw withering away, has pretty much done so now – and who will come marching over the border?

So there is plenty of life in the old procurement dog yet. And one caveat: the advent of open source makes the procurement process harder, not easier. Some footnotes on this are included in the main report; but procurement of open-source LMSs requires another as-yet-unwritten chapter.

1. Introduction

This survey was commissioned in May 2000 following a Higher Education Funding Council for England (HEFCE) circular inviting a series of expert surveys on e-tools for the e-University.¹ The work was funded by HEFCE and undertaken by Professor Paul Bacsich of the Virtual Campus Programme, Sheffield Hallam University, with specialist assistance from Professor Robin Mason (pedagogy), Dr Paul Lefrere (research and standards) and Peter Bates (reference sites and Internet futures).

The survey analysed over 40 specially commissioned vendor submissions as well as carrying out desk research on pedagogies of e-learning, e-tools research directions, reference sites and technological futures.

It offers a number of recommendations for appropriate ways forward in the light of expert analysis of this information.

Thanks are due to our colleagues in our own institutions, other agencies and sister studies, who gave of their time, usually at short notice, to answer questions where for reasons of confidentiality, speed and slowly emerging knowledge, we usually could not give them the full context. Special thanks are due to the vendor community around the world, who responded magnificently to the task of producing comprehensive reports on their products, to very tight deadlines.

2. Executive Summary

This describes the methodology used by the survey, changes to the methodology, and the conclusions reached.

2.1 Methodology

Because of the limited time-scale from May till July [2000], the project ran in four parallel strands:

1. 
   Pedagogic and assessment tools – systems.
   
2. 
   Pedagogic and assessment tools – reference sites (including content).
   
3. 
   Face to-face tutoring issues.
   
4. 
   Literature search, concentrating on new reports.
   

Strand a of the project was the most important aspect and had the following structure:

• 
  Creation of criteria for evaluation of these products and their usage. (This was mainly done in our response to the original tender.)² Criteria included system information (such as architecture, scalability, standards), user information (such as “industrial-strength” reference sites) and “futures” aspects of pedagogy and technology. The criteria were radically simplified compared with a full tender, since identification of trends, rather than selection of specific systems, was the goal of this exercise.
  
• 
  Identification of products and vendors to be considered (in consultation with sister e University studies, JISC – the Joint Information Systems Committee – and other advisors, etc.) – over 60 were contacted.
  
• 
  The sending out of a questionnaire to vendors inviting them to respond to the criteria. In fact there were two waves of questionnaires (see next section).
  
• 
  Checking by our team of the responses for internal consistency, and benchmarking them against our literature database.
  

Strand b identified key “e university/virtual university/virtual campus” reference sites of relevance to the UK e University, and reality checked vendor claims against these sites.

Strand c consisted of the preparation of a report on face-to-face versus online issues (section 10 of the main report.)

2.2 Changes to Methodology

In its response to the original tender, the study team stated that the original classification of systems suggested in the HEFCE tender document did not make good pedagogic or system sense. Consequently in agreement with HEFCE a broader survey was done.

During the project it was further agreed with HEFCE that this study should be somewhat extended to have general oversight of the other two e tools studies, cover any gaps identified, and look specifically at interoperability with administrative systems. This was one of the main factors that led to the second-wave survey of a further 35 vendors.

A draft final report was delivered to HEFCE on 27 June [2000]. The final report (of which this is the executive summary) incorporates feedback given on that report, phrases the conclusions in less technical language, fills in some gaps and incorporates information arriving over the summer; but comes to similar conclusions as the draft final report.

2.3 Conclusions

These are ten in number:

1. 
   Whatever the likely range of pedagogic strategies, business models and market demographics for the UK e University, there are e tools (software systems) that can deliver the required pedagogic and business strategies.
   
2. 
   For most approaches, there are a good variety of vendors; they are keen to offer solutions, and in some cases they have already have large-scale reference sites relevant to the e University.
   
3. 
   A critical mass of vendors is closely in touch with pedagogic issues – they often seem to be those who derive from, or are closely linked to, universities. Many other vendors are not in touch with such issues.
   
4. 
   At present only a few vendors have HEI sites in the UK making large-scale use of their products. It could be unwise to choose vendors who do not have a track record of large-scale implementation in situations similar to the e University.
   
5. 
   The UK university sector is now not the most advanced user of e tools in all ways – sectors such as FE, training and schools are in specific ways moving faster. Thus tools developed for those sectors – knowledge management, easy development of content, performance support and competence testing – may be relevant also to the e University.
   
6. 
   Most of the insights from research are steadily working their way through into products or pedagogic practice (among vendors who are early adopters).
   
7. 
   The e University should build on a basis of Web-based learning. There should be no overall dependence on storage media (such as CD-ROM and DVD) for delivery of e University courses; but these media will be relevant as an adjunct to the Web (as will text-books and face-to-face teaching).
   
8. 
   The issues of how, and to what extent, to move face-to-face teaching towards online teaching are now reasonably well understood by experts. One can expect to reduce the proportion of face-to-face teaching in the e University as technology advances and social conditions change.
   
9. 
   All e University systems should be oriented to supplying services to PCs. It is accepted that there are a number of “rivals” to PCs emerging on the market – interactive television, mobile devices, “Internet appliances”, etc. – but their penetration is still at the niche level in terms of relevance to e learning. Students of the e University should be assumed to have a PC of the general standard sold today for home or multimedia educational use, including a modem and printer. (Such PCs in the UK cost less than £1,000 including VAT.) PCs bought today should be able to adapt to higher-bandwidth connections, which will be available in the next year or two in some countries.
   
10. 
    All e University systems should be capable of delivering material in English and also the top 10 other languages in the world, measured in terms of likely student numbers for the UK e University.