Evaluation and Development Review Unit,
The Tavistock Institute, UK
This paper summarises the Tavistock Institute’s recent evaluation of the Electronic Libraries programme in the UK. The "Policy Mapping" methodology of the review is described, followed by some of the key findings. Cultural issues and futures are briefly discussed.
The programme is large and diverse. Established in 1994 and running through to 1998, it has an overall budget of £15 million. Significantly, for an evaluation exercise, the budget is spread across over 60 projects. These projects involve a greater number of institutions, as they are assigned to multi-institutional consortia. The programme as a whole is managed by a full time Programme Director, and his office has issued two calls for proposals.
The eLib programme results from the Follett Report [62], and can be divided into three parts:
This is further divided into the following ten programme areas:
The eLib programme was charged with implementing the relevant recommendations from the Follett Report, and to encourage developments by multiplier effects. It is this desire to foster learning, by catalytic and networking effects, which provides the rationale for funding many relatively small projects which, in aggregate, involve many institutions and individuals.
The programme is fortunate in having a full-time director. This appointment, which represents a significant investment, allows eLib to be managed in a coherent manner.
Another important aspect of eLib is its strong commitment to evaluation processes. It seeks to evaluate its work, and to use evaluation in order to learn from its successes and failures, and also to provide a tool for dissemination of eLib information.
The Tavistock Institute has evaluated eLib programme projects, as described later in this paper, using its established methodologies. In effect, it provided an expert resource to the eLib programme, to assist in assessing the effectiveness of the programme.
The evaluation approach can be used for ongoing services as well as for distinct projects. With ongoing services, the purpose is often to allow a process of continuous improvement. Feedback from users of the service being evaluated plays an important part in the approach.
Another rôle which a formal evaluation can play is in helping to ensure that value for money is obtained; and, correspondingly, that all those involved are accountable at the appropriate level.
The overall design of the programme is important, and may be mutually interdependent with the evaluation. A research and development programme must be structured so that it is open to the concepts of change and failure as much as to hoped-for successes. partly because formal evaluations can provide the mechanism by which some failures and some desirable changes are recognised, the evaluation can itself be formative. In this way, the evaluation process can play a key rôle in influencing the management, direction and planning of a programme.
The architecture used for the evaluation is shown diagrammatically in figure 1.
This shows the two principal components of the evaluation, namely the formative and summative evaluations.
Figure 1: Evaluation Architecture
In the case of the eLib evaluations, the self-evaluation guidelines were an important tool. The Institute provided guidelines for these evaluations. The self-evaluations were fed in to the projects’ annual reports.
The evaluation guidelines were very detailed. An example is shown in Figure 2, which lists the operational approaches applied to the Electronic Journals area.
|
General |
In Electronic Journals the main evaluation issues arise from the potential uses of multimedia journals by scholars in different disciplines and the attributes, facilities and systems these require. Although copyright issues and document format have been highlighted by many for investigation these represent just aspects of broader questions (respectively, the composition of entire editorial production, delivery, charging and payment systems, and the multi- media architecture of journals generally and in specific disciplines). Consequently, the rôle of evaluation in these projects is found in the attempt to learn how to produce multimedia journals and what multimedia journals are required, through developing actual or prototype electronic journals. |
|
Formative Rôle of Steering and Advisory Groups |
Receiving formative feedback on structure and (desirable) content. Raising expectations and gaining legitimacy among influential academics in the discipline(s). Involving and mobilising publishers. Developing models of negotiation and working between publishers and academics. |
|
Structured and Systematic Feedback |
Close analysis of reader and writer behaviour, experiences and reactions through:
Iterative translations of analysis of user behaviour/experiences/ reactions into functional specifications. Identification of use patterns (who? when? how often? what for? why not?) by online logs; traffic analysis; or surveys, for use in forecasting and planning. |
|
Forecasting and Planning |
Use of focus groups, modelling, scenario building, and other analysis to establish requirements such as:
Sustainability:
|
|
Common Elements in Annual Reporting |
General cognitive and communicative needs. Discipline-specific cognitive and communicative needs. Progress with:
|
Figure 2: Operational Approaches for a Sample Area
At the outset, we conducted an exercise to map the recommendations and policies emanating from the Follett Report to the actions taken by eLib. Policy Mapping has value in establishing a baseline for later evaluation of programme activities; for clarifying the strengths and weaknesses of a programme; and to provide an indication of desirable additional activities.
In brief, Policy Mapping involves starts with identifying and inventorying the policy objectives of the programme under study. These objectives are then matched, or mapped, to the actual activities of the programme; in the case of eLib the policy objectives were matched to the project activities. This is followed by the identification of limiting factors, and determination of the scope for future action.
For this exercise, we applied three techniques:
Our overall finding was that there are no significant areas mentioned or implied the by Follett Report, or indeed in the wider literature, uncovered by some HEFC library-related initiative. This good news contrasts with situations in other initiatives, where we often find that practice differs from the intended policy.
Some of the lessons common to most projects were the high values of:
The consortium structure provided both strengths and weaknesses. A key strength is in the way the structure provides in-built "networking", as well as assisting in dissemination, as described above. On the other hand, distributed consortia are difficult to manage effectively, for reasons which include different allegiances and geographic dispersal.
There is a rich variety of electronic journal projects. Some of the journals are experimenting with non-traditional media (i.e. other than text and static images), and the basic notions of journals are being deconstructed, as new environments are experimented with. These projects provide a broad basis for possible future developments. However, in general they exhibited a degree of organisational naïveté. This arises from a failure to recognise the reality of running a journal; some projects are now discovering just complex the required commercial activities can be.
Likewise, the work in the area of on-demand publishing has given rise to a number of experimental services, but all these will require much more work if they are to be integrated with teaching strategies, library reorganisation and publisher requirements. Also, more work is needed on questions such as intellectual property rights and funding.
Work on document delivery raises questions about the future developments and about the business case.
The most stable and coherent part of the eLib programme is that which deals with access to networked resources. However, this calls for further evaluation; the current discipline-based model may not be the only or best approach.
Finally, much good work has been done on training and awareness. However, it would be more beneficial if future efforts were related to commissions rather than to calls.
eLib is not only about information technology. It is at least partly about cultural change, in other words how we are organised to use the developing IT tools.
Cultural change involves new frames of reference, new ways of acting. It results from actors acquiring new symbolic resources (such as cognitive frames and paradigms, concepts, knowledge and skills) in changed structural contexts where these symbolic resources are meaningful, deployable and operational. Cultural change is as much the result of structural change as a promoter; effective cultural change rarely happens without simultaneous change in structures.
Ideas of cultural change will be important to the future of the eLib programme. More work is needed on the twin areas of training and awareness, and similarly on communication and dissemination. A strong organisational commitment is needed to ensure successes; and care is needed to match the scale of projects to the corresponding critical masses.
Finally, a degree of integration and convergence is desirable, as the successful projects grow towards each other into a successful whole providing overlapping sets of services for overlapping disciplines.
[60] This account was drafted for this report by The Marc Fresko Consultancy. It is based on notes taken during the presentation and slides used by the speaker.
[61] eLib, the Electronic Libraries Programme. See url http://www.ukoln.ac.uk/services/elib/
[62] Joint Funding Council's Libraries Review Group: Report, 1993, available from http://www.ukoln.ac.uk/papers/follett/report/