Evaluation of the Electronic Libraries Programme
Synthesis of Annual Reports


In providing an overview of the progress of eLib projects during the first year of the programme, we focus primarily on the generic activities in which they have all been involved and which to a large extent characterise the start-up phase of any innovation project. Within domain clusters, projects were also engaged in similar activities and we briefly outline these in Section 2.2, Domain Specific Activities, below.

2.1 Generic Activities

For the large majority of projects, the period covered by the first annual report was primarily concerned with getting the project up and running. For a small minority, eLib funding supported an already established or embryonic activity and so these projects were able to move more rapidly towards implementation. Nearly all the first round projects had progressed beyond the start-up phase to implementation and the achievement of project milestones, although a small minority experienced continuing problems with staffing and technical equipment which delayed their progress.

The activities during this start-up phase can be grouped as follows:

2.1.1 Staffing

Appointing new staff and assembling a project team was a primary activity for all projects, readily accomplished by some but a source of ongoing difficulty for others due to difficulties of recruitment and staff turnover. This reflects the scarcity of the particular skill profile required by eLib projects and consequent high marketability and labour market mobility of such persons.

The composition of project teams or staff groups varies widely as does their mode of functioning. The models of staffing adopted by projects include varying combinations of the following personnel:

Although most projects have a small number of paid dedicated project staff (most commonly between three and five), the project team size is augmented by these other categories of personnel who in many cases greatly outnumber staff appointed to project posts. Some project teams have as many as ten to thirteen members, distributed across different partner institutions and organisations.

A common staffing pattern includes the following role-holders and areas of expertise:

Depending on the area, the project team might be supported by teams or individuals in partner institutions developing courseware materials, delivering training, undertaking editorial work, providing an operational service. Much of this work is undertaken on a voluntary basis, underlining the success of eLib in leveraging additional resources.

Several projects have already reviewed their staffing profile, in some cases deferring appointments to a later stage in the project life-cycle, in others re-combining roles and in yet others identifying the need for quite different kinds of skill. Evaluation is an area where several projects feel they lack appropriate expertise and will need to redirect resources to this area.

The annual reports draw attention to the quality, professionalism and enthusiasm of the personnel involved in making their project a success, whilst alerting eLib managers at the same time to the substantial unpaid time input which is unlikely to be sustained in the longer term and to the evident under-staffing in some projects. The latter tends to be compensated for by over-commitment of those working on the project, already leading in some cases to burn-out and consequent staff turnover.

2.1.2 Management structures and processes

Virtually all projects have set up a Steering Committee of some kind (variously named Project, Advisory or Management Board, Project Advisory Group, Executive Committee, Steering Group) which meets at regular intervals, usually quarterly or half-yearly.

As one might expect from such diverse projects, the actual composition and size of Steering Committees varies. Not all annual reports included information about membership, however, so our overview is partial. In several projects, the membership is exclusively drawn from academic and library staff in universities who are usually representative in some way of the intended user base; others have additional members from relevant professional associations and key organisations such as the British Library; a smaller number have a more balanced membership that includes publishers and other commercial organisations. As required by the conditions of funding, project steering committees include FIGIT representation.

There is relatively little information in the annual reports about the functioning of steering committees. Our impression is that projects primarily look to their steering committee for advice on policy and overall strategy although in some instances the members are involved in a hands-on way.

The Tavistock Institute’s Evaluation Guidelines for eLib Projects emphasises the importance of a Steering Committee with membership and function strategically chosen to enhance uptake and exploitation of project outputs and learning. Some projects have clearly been strategic in their choice of members whilst in others it would appear that membership is based more narrowly on familiar networks. One conspicuous input missing in many projects is that from university senior management. Occasional presentations to senior management are likely to be important for the longer term integration of eLib innovation beyond the confines of the library and participating individuals and departments.

The most common arrangement for project management is a project team, overseen by a project director and with operational responsibility vested in a project manager or coordinator. Variations on this arrangement include a project executive group supported by standing ‘working parties’ or ‘project assurance teams’ with different areas of responsibility. In the Electronic Journals area, an Editorial Board often sits alongside the project management group.

Project management is complicated by the dispersal of staff across different institutional sites, and by the varying combination of dedicated, part time, and voluntary or unremunerated staff. At a purely pragmatic level, finding a suitable date for a meeting can itself be quite problematic. One or two reports commented on the time it takes to build effective modes of communication and decision-making and to establish protocols and routines that underpin smooth management. Most projects combine periodic face-to-face meetings of the project team with one or more modes of electronic communication, including occasional use of tele- or video- conferencing. In a number of cases, projects had hoped to use e-mail as a substitute for meetings but found this arrangement unsatisfactory.

2.1.3 Technical and legal infrastructure

Installing and customising technical hardware and software has been a pre- occupation of most projects, particularly those dealing with applications that use advanced rather than stable technologies or which call for complex systems integration. Many projects have experienced difficulties in establishing the basic technical infrastructure, commonly as a result of delays in delivery of equipment, missing parts or faulty software. Further set-backs have arisen from projects’ dependency on wider institutional technical infrastructure, including technical advice and support services, which is often inadequate for the functionalities being developed.

Specification of the systems architecture, including investigation of technical choices and the production of technical documentation, has been an early start-up task. Project staff have also had to familiarise themselves with new hardware and software and learn the new skills associated with designing Web pages, the production of PDF files, the marking up of HTML, etc.

On the legal side, several projects have developed model agreements or memoranda of understanding between the partner organisations and in one case the project has established itself as a legal entity. Others, however, have preferred to function with more informal arrangements between partners. Additional aspects of legal infrastructure include making provision for indemnity against copyright infringement by authors as well as the installation of security systems.

2.1.4 Project planning and review

The start-up activities of projects have often included the development of a work plan, and the elaboration of an evaluation plan that takes account of the Guidelines on Project Evaluation and the evaluation workshops and clinics. Far fewer projects have developed a business plan for the downstream uptake and exploitation of products and services, although projects have been encouraged to plan for sustainability at the outset rather than as an end-phase activity.

A small minority of projects brought the partners together early on to review the project concept, to clarify how far they have a shared understanding of the innovation and to make explicit what partner organisations hope to gain from their involvement in the project. Where this has occurred it has been seen as a fruitful activity, although there are other projects which clearly would have benefited from this degree of collective reflexivity in the early stages of the project. Several annual reports have made reference to difficulties in communication between the partners that reflect different expectations or understandings as well as different modes of discourse.

2.1.5 Promoting the project

Awareness-raising has been a significant area of activity for all projects. In general, promotional activities are focused on some tangible product or service that is the intended outcome of the project, and are aimed at creating a receptive climate for their innovation and at enhancing the prospects for user acceptance. The broad strategic actions adopted by projects as part of general awareness raising and project promotion are generally multi-pronged, including some combination of the following:


All projects have developed promotional material and engaged in a variety of activities aimed at raising the profile of their project. Profiling has generally included the adoption of logos and creation of a distinctive identity but has also extended to merchandising. The most common activities reported by projects in descending order are:

Projects have also been active in setting up electronic communication tools as an awareness raising device including Web advice point, newsgroup archive, e-mail discussion list and closed list on Mailbase.


Training events have also been used by projects as a strategy for awareness raising. Participants at these events are introduced to Web and Internet use and to the benefits that the project innovation has to offer in an electronic environment.


Projects make use of existing networks as a means of promoting the project and raising awareness about developments in electronic libraries more generally. Many projects adopt a cascade model of networking, focusing on the gatekeepers, people with status and credibility within the professional or academic discipline and those in pivotal positions, in the expectation that with their active support and endorsement the project will more readily gain user acceptance and diffuse more widely.

2.1.6 Continuous formative evaluation

Evaluation, in the sense that we use it here, is integrated into a user-centred development process in which feedback from the user population informs the design and ongoing development of the software application in an iterative way.

Key features of a user-centred approach are:

1. knowledge of the real users or the potential user population and the context in which the innovation is being introduced

2. feedback from users at key stages in the development lifecycle that informs the ongoing development of the innovation

3. involvement of key stakeholder groups, including the various user communities, in decision-making about design choices

4. strategies for empowering users so that they can exercise a genuine user influence over the systems development process

Whilst ‘best practice’ in systems development is moving towards a coherent user-centred design process incorporating all the above, more generally users are seen as sources of feedback rather than active participants. The annual reports present a similar picture for eLib systems development practice: there are some notable examples of projects where the end users have a real stake in the systems being developed and are empowered in the dialogue with developers, but the more general picture is one of involving users primarily as sources of feedback. It is important to acknowledge this widespread commitment to iterative feedback processes as exemplars of good practice, even if not yet matching up to best practice in the eLib programme.

Only a few projects made any mention of a systematic profiling of their intended users or an analysis of the context in which their product or service is to be used, whether of the intended future user population or more narrowly of their pilot sites. In those cases where data on the user population was gathered, the methods used were fairly simple ones:

In relation to pilot sites, three projects referred to the nature of their information gathering activities, these included:

Many more projects however have gathered feedback from users on such aspects as the user interface, user requirements, reasons for non-use or take-up of the facility, and areas for improvement. The methods are quite diverse, some of them being one-off surveys at a fixed point in time, others of them gather data on an ongoing basis, and yet others entailing a distinctive structured arrangement for associating users with the development process.

The OMNI Approach to User Centred Development

The OMNI projects exemplifies a user-centred approach, formalised through a Senior User, a Project Assurance Team, Advisory Groups, Volunteers and Focus Groups:

"Right from the start, it was felt that user involvement needed to be established throughout the development of the service. This is especially highlighted by the inclusion of a user representative (the Senior User) in the OMNI Project Board and Executive Board, who has advised on all aspects of OMNI development throughout the life of the project.

Additionally, a Project Assurance Team (PAT) was convened, which includes representatives from two key user constituencies for OMNI: the University Medical School Librarians Group and the University Health Sciences Librarians, as well as the Project Researcher from SOSIG. Subsequently, the PAT has been broadened to include an additional volunteer member from the user community and also a nominee from the Centre for Information Quality Management. The remit of PAT has been to supply independent preview and review of OMNI user-oriented products, primarily the user interface, and resource descriptions, user documentation, and presentations and workshops.

Two key areas were additionally identified in which it was felt that OMNI could benefit from direct contributions: the development of evaluation criteria by which OMNI staff and volunteers could assess the value of network resources for inclusion in the OMNI database; and issues relating to indexing and classification of resources selected for inclusion. Consequently, two Advisory Groups were convened to support the work of the OMNI team in these areas (email discussion lists were specially created).

The volunteer effort has been a cornerstone of OMNI activity, primarily aimed at information professionals, although practitioners have also participated. Two Volunteers’ Workshops have been held, designed to address volunteers’ special training and awareness needs. The OMNI-collaborators Mailbase list supports OMNI contributors and offers a useful platform for the discussion of issues that arise.

Finally, in order to listen to users directly, focus groups have been planned with three user constituencies."

The methods we came across in the annual reports included:

It is not clear from many annual reports how the data gathered are fed back into the development process, or whether users have any input into the key design choices as distinct from general usability aspects. A small number of projects are however explicit about these links, reflecting their mature user-centred design approach. In two cases, the elected chair of the User Group is represented on the project Steering Committee; in another, the user survey working group is one of four working groups that feed directly into the project executive and steering committee.

2.2 Domain Specific Activities

2.2.1 Electronic journals

The specific activities undertaken by projects in the EJ domain are clustered in ways that reflect the different purposes of EJ innovations. Those projects that seek to establish an electronic journal in a disciplinary area have been engaged in a similar set of activities which include:

Electronic Journals

Demos, prototypes & trials [ Not necessarily a comprehensive listing. ]

First issue of Sociology On-Line, two issues of Journal on Information Technology and Law published

Two trials of Reviews in History, three trial Transactions papers

First issues of Internet Archaeology ready for publication

Eleven Electronic Seminars in History established

Demonstrator user application, using Isite and BRS as search engines, developed for SuperJournal

Web server, with evolving interface components, installed to host project materials in Open Journals

Web site supporting abstracts and PDF files from two journals set up in Electronic Journal and Learned Societies

Some notable achievements

‘Consolidation of a national network covering an entire discipline’ (Reviews in History)

‘Publishers, in particular, are learning and moving forward in their thinking due to the project’s analysis and technical development work’ (SuperJournal)

‘First paper on-line demonstrates the viability of the concept and indicates the variety of possible presentational techniques’ (Internet Archaeology)

‘Significant new contacts with publishers who share the vision’ (Open Journal)

2.2.2 Access to network resources

Projects in this domain are engaged in a similar set of activities, although the established nature of networked resources in some disciplinary areas has allowed some projects to focus more on refining the interface and facilitating the creation of new networked resources than on developing the services specification and technical capabilities of the service. One project, dealing primarily with grey material in a politically sensitive, has found considerable copyright problems. The common ANR activities of projects include:

The particular successes reported by projects reflect their individual priorities and identities. In broad terms, successes in this first year relate to:

• technical development, including for example the shift from a general purpose WWW server software to a custom-server optimised for delivery of template database

Access to Network Resources

Demos, prototypes & trials [ Not necessarily a comprehensive listing.]

Piloting of the Edinburgh Engineering Virtual Library in six test sites

Launch of the OMNI medical database with over 700 resource descriptions, and seminar for 170 delegates

Demonstration of the ROADS database software in SOSIG, the social science gateway, with over a 1,000 quality resource descriptions

Launch of ADAM’s prize winning descriptive Web site

Some notable achievements

‘1,000 UK accesses per week achieved within three months’ (IHR-Info)

‘Impressive response to training workshops and materials in use of Internet for social scientists’ (SOSIG)

‘Early provision of software for stable and well designed subject gateways.....Major participation in, and stimulation of, standard making activities’ (ROADS)

2.2.3 On-Demand publishing

The ODP projects span a range from the more technical, experimental applications to those primarily concerned with the implementation in real environments of ODP courseware based on low cost stable technologies. Although most projects are engaging in a similar set of activities, the prioritisation among them reflects their positioning on the experimental-implementation spectrum. Common activities are:

The achievements reported by this group of projects relate as much to ‘understandings’ gained as to successful outputs or outcomes. Among those mentioned are:

• evidence of a rethinking of strategies towards course pack provision by suppliers of document management systems

On Demand Publishing

Demos, prototypes & trials [ Not necessarily a comprehensive listing.]

Open Learning Foundation materials in two subject areas mounted on eOn server

Creation and field testing of Web mounted electronic course booklets for three Spring semester modules on Media, Critical and Creative Arts

Production of five SCOPE course packs for undergraduate sociology students at two Scottish universities, involving negotiation with nearly 60 rights holders

Some notable achievements

‘Research undertaken with suppliers of document management systems resulted in suppliers rethinking their own strategies towards course pack provisions’ (Phoenix)

‘Development and implementation of a ‘four stage’ security system, while not making illegal use impossible, is nonetheless recognised by publishers as a reasonable compromise between ‘free for all’ and technical restrictions which undermine the advantages of electronic materials’ (ODP in the Humanities)

2.2.4 Digitisation

The two digitisation projects included within this overview were dealing with different kinds of materials that raised different copyright issues. Even so, there was a similarity in the activities undertaken during the first year:


Demos, prototypes & trials [ Not necessarily a comprehensive listing.]

Creation of a trial CD-ROM to facilitate user testing of digitisation in the field of Art and Design

Trial digitisation of over 200 successive pages of the 19th Century journal, Notes and Queries, and resulting files mounted on Web server

2.2.5 Electronic Document Delivery

The projects in this domain vary considerably in their stage of development, one or two being well advanced with field trials whilst others are still developing the technical configuration. At least one project is still seeking clarification of its overall concept and logic. The activities of the project in the first group includes:

The projects which have not yet arrived at an implementation phase are involved in such activities as:

None of the EDD projects reported any ‘noteworthy achievements’ as yet in their annual reports, though where field trials have taken place this may be seen as an achievement in itself.

Electronic Document Delivery

Demos, prototypes & trials [ Not necessarily a comprehensive listing.]

Eurotext now contains 27 titles on the topic of Environmental Policy

Trial of the prototype LAMDA service in nine participating institutions, followed by an extension to a wider group of 15 request only libraries

2.2.6 Training and Awareness

The T&A projects adopt a wide range of strategies for changing the attitudes and practices of librarians and other actors in the higher education system. Those employing a training strategy have prioritised training needs analysis or information gathering followed by the development and piloting of course materials and workshop events as key activities. Other activities are specific to the individual projects and include:

Training and Awareness

Activities and initiatives [ Not necessarily a comprehensive listing.]

Design and delivery of the Edulib 16 workshop programme

80 Netskills awareness raising events with over 3360 people

Identification of NetLinks co-ordinators in 23 institutions, followed by NLS institution reviews and 19 focus group discussions

Four issues published of the parallel print and Web magazine Ariadne

Some notable achievements

‘We were requested to increase our print run substantially after the appearance of Issue 1, while the Web version has grown in volume and variety as we have attracted a wide range of contributions from the UK HE community and beyond’ (Ariadne)

Table of Contents

[ Top of Page ] - [ Up ]

The Electronic Libraries Programme (eLib) was funded by the Joint Information Systems Committee (JISC)
Page version: 1;
Web page maintained by UKOLN Systems Team and hosted by UKOLN - feedback to systems@ukoln.ac.uk .
Site last revised on: Tuesday, 24-Nov-1998 14:21:04 UTC
DC Metadata

Last updated 04-Apr-1997