UNIVERSITY OF CENTRAL LANCASHIRE
CENTRE FOR RESEARCH IN LIBRARY & INFORMATION MANAGEMENT

MANAGEMENT INFORMATION FOR THE ELECTRONIC LIBRARY (MIEL) PROGRAMME

Management Information Systems and Performance Measurement for the Electronic Library:
eLib Supporting Study

(MIEL2)

Final Report

by

Peter Brophy and Peter M. Wynne

June 1997

Publication details

© JISC, 1997.

Acknowledgements

The authors wish to acknowledge with gratitude the assistance of many expert colleagues in the preparation of this work. We are particularly grateful for the input of the participants in the Expert Workshop held on 16th May 1997 in Preston, for that of Professor Charles McClure who provided an update on American work in this field and for the time given by members of SCONUL’s Advisory Committee on Performance Indicators (ACPI). Finally we would wish to express our thanks to colleagues in CERLIM, and in particular to Shelagh Fisher, for their suggestions, advice and assistance.

PB
PMW

Preston
June 1997

Contents

Executive Summary	
1. Background	
1.1 The Electronic Library	
1.2 Performance Measurement	
1.3 The Study Team	
1.4 Related Work	
2. Methodology	
3. Performance Measurement Model	
4. Related work	
4.1 Van House, Nancy A., Weil, Beth T. and McClure, Charles E. Measuring academic library performance: a practical approach.  London and Chicago; American Library Association, 1990.  ISBN: 0-8389-0529-3.	
4.2 King Research Ltd. Keys to success: performance indicators for public libraries.  London; HMSO 1990.  ISBN: 0-11-430048-8.	
4.3 Ward, S. et al.  Library performance indicators and library management tools. Luxembourg; European Commission DGXIII-E3, 1995 [EUR 16483 EN].	
4.4 International Standards Organisation ISO 11620: Information and documentation - library performance indicators. Working draft 10. ISO Secretariat ISO/TC 46 SC8/WG4, 1995.	
4.5 IFLA.  Measuring quality: international guidelines for performance measurement in academic libraries. IFLA Section of University Libraries and Other General Research Libraries.  Saur, 1995.	
4.6 Commission of the European Communities, Fourth Framework (Telematics), Libraries Programme: Project EQLIPSE (Evaluation and Quality in Library Performance: System for Europe) and Concerted Action CAMILE (Concerted Action on Management Information for Libraries in Europe).	
4.7 McClure, Charles R. and Lopata, Cynthia L.  Assessing the academic networked environment: strategies and options. Coalition for Networked Information; 1996.	
4.8 East, Harry, Sheppard Elaine and Jeal, Yvette.  A huge leap forward: a quantitative and qualitative examination of the development of access to database services by British Universities, 1988-1994. London; British Library Board, 1995. [British Library R&D Report 6202].	
4.9 HEFC(E)  The effective academic library: a framework for evaluating the performance of UK academic libraries: a consultative report to HEFC(E), SHEFC, HEFC(W) and DENI by the Joint Funding Council’s Ad Hoc Group on performance indicators for libraries. Bristol; HEFC(E), 1995.	
4.10 Other Studies	
5. Performance Indicators for the Electronic Library: I Operational Management	
5.1 Introduction	
5.2 Indicators for Operational Management	
5.3 Issues	
5.7 Validation of Indicators	
6. Performance Indicators for the Electronic Library: II Forward Planning	
7. Performance Indicators for the Electronic Library: III Evaluation and Review	
7.1 Integration	
7.2 User Satisfaction	
7.3 Delivery	
7.4 Efficiency	
7.5 Economy	
7.6 Conclusions	
8. Conclusions	
9. Recommendations	
9.1 Testing of the proposed indicators	
9.2 Agreement on the core set of indicators	
9.3 Modelling the electronic library	
9.4 Agreement on international standards	
9.5 Collecting data	
9.6 Understanding the operation of electronic libraries	
9.7 The Impact of the Electronic Library	
10. Contact Details	
Appendix 1: Report of the MIEL1 Scoping Study	
Appendix 2: Complete list of indicators for evaluation and review (EAL +)	
Appendix 3: MIEL Expert Workshop	

Executive Summary

The Management Information Systems and Performance Measurement for the Electronic Library project was part-funded by JISC under the Electronic Libraries Programme (eLib) as a Supporting Study. The contractor, the Centre for Research in Library & Information Management at the University of Central Lancashire, undertook the work as part of its Management Information for the Electronic Library (MIEL) Programme having previously undertaken a Scoping Study for eLib. 

This Final Report begins by reviewing current trends in library development, with particular attention to emerging understanding of the electronic library’s role in institutions. The concepts of Resource Discovery, Resource Delivery, Resource Utilisation, Infrastructure Provision and Resource Management have taken on new emphases as attention has shifted from local collections towards access to the global information networks and content. However, work on performance indicators, and more broadly on library management information, has not kept pace with these developments. A review of the major work in this latter field has identified a series of studies which bear upon the subject, but only McClure’s work in the United States can be said to be of direct relevance to electronic as opposed to traditional libraries.

Following on from the work and recommendations of the Scoping Study, the core of this Report is structured around a threefold approach to library management: the needs of operational managers for day-to-day management information is considered first; then the need for planners for information which enables a forward look to be taken; finally, the strategic manager’s need for information and indicators which bear upon evaluation (i.e. by definition, ‘the value of’) and review. In this last category the Report follows closely the structure of the HEFCs’ publication, The Effective Academic Library.

While it has proved possible to adapt many existing indicators to the needs of the electronic library, it has been necessary to adopt different approaches in some situations. Thus, in the first category, the recommendations of The Effective Academic Library regarding integration between the library’s services and the academic work of the institution can be expanded readily to incorporate electronic services - although not without raising some important issues. For example, it is far from clear how the boundaries of the electronic library should be defined when it is not limited by a physical building or buildings.

In defining other indicators, however, a different approach must be taken. The most obvious example of this occurs with the development of indicators which are the equivalent of the traditional library’s measurement of ‘numbers of documents delivered’. A considerable amount of effort was expended during the study in trying to devise indicators of electronic library effectiveness based on counts of electronic documents. After discussion with many experts, and in particular after the Expert Workshop which was held during the Study, we determined that this approach should be abandoned. In an electronic environment not only is it virtually impossible to define a ‘document’, but the key issue for users is not the number of documents they can download but the range and depth of resources which are available to them. It is this concept which has been developed in the Report.

The indicators chosen to supplement The Effective Academic Library have been identified in an Appendix, which lists the expanded set (termed EAL+) - see page 84.

The Report ends with Conclusions and Recommendations for further work. In particular it is important that empirical work is undertaken to test and validate indicators in the context of operational electronic academic libraries.

1.	Background

1.1 The Electronic Library
Libraries, and academic libraries perhaps most of all, are undergoing rapid change as they come to terms with, and seek to exploit for their users, the networked information environment. In less than two decades they have moved from the use of computers for what was called ‘housekeeping’ through mediated exploitation of commercial ‘on-line’ services to the present position where digitised information is an integral part of their stock-in-trade. In the UK, the Follett Review galvanised librarians, vice-chancellors and the funding councils into action. It led directly to the Electronic Libraries Programme (eLib) and to the requirement on institutions to produce ‘information strategies’. Current initiatives such as the exploration of the need for a ‘National Agency for Resource Discovery’, the British Library Research & Innovation Centre’s Digital Library Research programme and JISC Calls for Proposals on ‘authentication, ‘clumping’ and ‘hybrid libraries’ demonstrate the rapid progress which continues to be made.

As yet there are few clear models available on which the strategic vision of the electronic library can be built. The term ‘electronic library’ is itself controversial, and some prefer to call it (whatever ‘it’ may turn out to be!) a ‘virtual library’ or ‘networked library’. Others regard the whole debate as sterile, since the whole concept of a library needs to be revisited in an environment of readily available, globally networked information. Such issues will continue to engender heated debate for many years to come.

Some of the issues are, however, becoming clearer. If we assume that the principal role of the academic library is to enable its users to identify, locate, gain access to and use the information they require, then the ‘library’, whether traditional, electronic or hybrid, may be identified and characterised by a series of five functions:

	- it provides tools which enable users to view the ‘electronic information landscape’, through the sources of metadata which have been provided either directly by the library or to which it provides access. This is a ‘resource location’ or ‘resource discovery’ process. It may be seen as a two stage process of resource identification and location identification i.e. the user may identify a resource and then identify a location which holds it. The second locating process may be left to the library to perform (as when a user completes an interlibrary loan form but does not specify a holding location);

	- it provides tools which enable users to gain access to the information which they have identified as being of interest to them. This is a ‘resource provision’ or ‘resource delivery’ process. In some cases resource delivery may be a three-stage process, whereby the user, having discovered an item, first requests it, the library then acquires it, and the library then delivers it to the user - this is the classic traditional library process expressed, for example, in a reservations or inter-library loan service. In an electronic context these processes are usually concatenated;

	- it provides tools which enable users to exploit the information content to which they have been given access. This is a ‘resource utilisation’ process. In an electronic context the tools will include word processing, spreadsheet and database software together with filters and specialist display software such as Adobe Acrobat;

	- it provides, possibly through third parties, the physical infrastructure and support services which users need to exploit information resources. This is the ‘infrastructure provision’ process. One of the functions of the electronic library will be to provide network infrastructure, PCs, printing facilities and so on. It will also provide support in the form of helpdesk and advisory services;

	- it provides management structures and procedures which ensure that the resources available to it are used to provide the maximum possible value for money to its users. As part of the management function, decisions will be taken on which resource discovery tools to provide, on how specific resources should be delivered (for example, should they be held locally or accessed from remote servers) and on which tools should be provided to enable users to exploit the information. In addition, management will provide procedures to handle the economic and legal aspects of information provision. Together, these may be described as ‘resource management’.

These functions are shown diagramatically in Fig. 1 below:

Resource Discovery	Resource Delivery	Resource Utilisation	Infrastructure Provision	Resource Management	
Resource identification
Location identification	Request
Acquire
Deliver to user	Exploitation tools	Space
Equipment
Networks
Support services	Prioritisation
Value for Money	
Fig. 1

Although these functions can be found in the traditional library, in the electronic environment they introduce new levels of complexity, not least because so many of the information resources are not ‘owned’ by the library in any real sense: issues such as cataloguing of networked resources are therefore far more complex than their traditional library equivalents.

None of the functions described above assumes that the library acts as an intermediary: increasingly it must be expected that academic information resources will be discovered by and delivered to the end user. It follows that, once networking becomes pervasive, the concept of the library as ‘place’ becomes essentially meaningless, except insofar as (a) the library operates as a ‘hybrid’ of electronic and traditional services, and (b) one of the ‘tools’ provided is a PC and a place to sit (and maybe drink coffee!). In the academic environment there are likely to be pressures to deliver more and more courses of study off-campus: the electronic library is ideally placed to support this once the conceptual focus is shifted away from the geographic location.

Teaching and learning in higher education are increasingly moving towards ‘student-centred’ and ‘problem solving’ approaches in which the student, often working in a group, is encouraged to explore a problem or issue and to find his or her own solutions. Such approaches demand the ability to explore the literature and find pertinent information of a kind which cannot be pre-determined through reading lists and the like. The networked environment is ideal for this kind of approach, and again the electronic library will provide an opportunity to support such developments.

Similarly, the research process is changing as a result of the ability of researchers to communicate with one another, and to view experiments, at a distance and virtually instantaneously. A research team can thus contain the experts in the field almost regardless of their physical location, and certainly regardless of national or institutional affiliation.

However, one of the effects of changed approaches to learning, of the globalisation of research, and of the networked information environment is that intra-institutional boundaries become blurred. It is already difficult to define the boundaries between the ‘university library’ and the ‘computing services’, and many institutions have converged their academic support services in recognition of this. It may be that an equal loosening of boundaries between academic departments and support services will be experienced in the future, especially as sophisticated multimedia packages and communications and conferencing software become commonplace in teaching.

The significance of these developments for the library manager operating in an increasingly electronic environment are profound. Traditional approaches to library management have been developed in response to challenges of organising essentially local collections. To operate in a institution-wide, national and global context poses new challenges and will require new management tools. This Report, but more generally the Management Information for the Electronic Library (MIEL) Programme of which it forms a part, is a contribution to that development. As we report in our conclusions, however, there is still a very long way to go, in a constantly changing environment, before anyone will be able to claim that a comprehensive management framework has been put in place.

1.2 Performance Measurement
The importance of performance measurement has been recognised by librarians for many years. The first systematic studies in the UK were carried out in the 1960s, and included public library issues and analyses of academic library management decision making. Even earlier studies of the scientific literature had provided a systematic basis for decisions on periodical stock.

In recent years interest in performance measurement has been intense and a variety of studies have been published on both sides of the Atlantic. The reasons for this interest are not hard to find: pressure on resources has led to an ever-more intensive search for efficiency of operation, while a concern to serve users’ needs has focused attention of effectiveness. Funders have not only demanded that value for money be achieved but that it be demonstrated by reference to factual data. Users, and other stakeholders, have become more vociferous, while the adoption of “access” in preference to “holdings” strategies has led to greater reliance on external providers and with it greater use of contracts, service level agreements and the like.

The development of generally accepted and reliable performance indicators for UK academic libraries was given impetus by the Follett Report and subsequently work was undertaken by the Joint Funding Councils Ad-Hoc Group on Performance Indicators for Libraries, resulting in the publication of The Effective Academic Library (referred to throughout this Report as EAL). EAL drew on a wide range of sources , including CURL, Conspectus, work by HEQC, IFLA’s work on international standards and that of ISO, EC activity and work at Liverpool John Moores University using methodologies developed by Van House. It should be noted that work on EAL continues, with a British Library funded study at Cranfield University exploring some key issues concerned with its implementation.

Of immediate concern, but of even greater importance as we look to the future, is the lack of defined performance indicators for the electronic library. A Scoping Study, which reported in 1995, brought together leading experts and agreed a framework for development in this area and identified promising lines for research. The present Study builds on that Scoping Study to provide definitions of performance indicators suitable for use in the management of electronic library services, and identifies a series of issues which need to be taken into account in this area, as well as indicating where it is felt that further work will be needed.  

1.3 The Study Team
The University of Central Lancashire’s Centre for Research in Library and Information Management (CERLIM) has a long-standing interest in performance measurement and management information systems, as well as considerable experience in the development of leading-edge systems for the electronic library. CERLIM carried out the scoping study which is referred to above. A particular feature of CERLIM’s work in this area has been an emphasis on qualitative measurement linked to assessment of user satisfaction, arising from a recognition that all library services are put in place to serve customers. A recent British Library funded study has reinforced this approach.

CERLIM undertakes a broad range of research and development related to libraries and information management. It is well-known internationally, especially for its leadership of a series of European Commission funded Telematics projects under the Libraries Programme. More broadly it is involved in a number of eLib projects and studies, is active in database development both for the UK Department of Health and DGV of the European Commission and has been funded by the British Library Research & Innovation Centre for a variety of projects. Permanent staff of the Centre are assisted by a number of secondments and research students. Staff also undertake a range of consultancy activities.

The University of Central Lancashire’s Library & Learning Resource Services (LLRS) provide a good operational model for work in this area since they operate both a distributed and converged service, providing nineteen delivery points spread across the North-West of England, exploiting the electronic library to the full via pervasive networking, while also operating a major, traditional library service at the main site in Preston.

The Management Information for the Electronic Library Programme is currently conceived as operating through five distinct phases or projects:

	- MIEL1, the Scoping Study referred to above;

	- MIEL2, the Study which is the subject of this report;

	- MIEL3, international standards activity designed to provide agreement on indicators and paralleling work on traditional indicators by such bodies as ISO and IFLA;

	- MIEL4, examining the issue of management information requirements in co-operative networked electronic environments such as clumps and hybrid libraries;

	- MIEL5, researching the management information needs which arise when electronic libraries are delivered to dispersed and remote populations.

Although the focus of the Programme is on academic information services and libraries, CERLIM is also active internationally in work on the management information needs of other types of library, including public and national libraries.

1.4 Related Work
In undertaking the Scoping Study, CERLIM took particular note of work taking place outside the UK and made contact with key researchers. The work of the ISO TC46/SC8 Sub-Committee which has led to the acceptance of ISO 11620 (Information and Documentation: Library Performance Indicators), IFLA’s work (which is co-ordinated by Dr Roswitha Poll of Universitäts Münster, one of the EQLIPSE project partners), the EC-PROLIB study undertaken by De Montfort University, and SCONUL’s Effective Academic Library work with the HEFCs was all considered. Of particular relevance was the work by Charles McClure and Cynthia Lopata of Syracuse University in the USA, subsequently published as Assessing the Academic Networked Environment: Strategies and Options, which was also discussed at the November 1995 CNI conference in Portland, Oregon, at which CERLIM was represented. All of these sources remain relevant. In addition, discussions in JASPER and particularly a series of reports on the use of JISC Services produced by Harry East and his colleagues at City University have emerged in the intervening period.

2.	Methodology

This Supporting Study was undertaken using the following methodologies:

	- desk research, using information sources identified in the scoping study, with a particular emphasis on the EAL work. As reported in Chapter 4. This included analyses of the major performance indicator studies which have been published during the last ten or so years. The Study Team was particularly concerned to analyse in detail the work undertaken for UK academic libraries since the publication of the Follett Report and the ground-breaking work currently being undertaken in the USA by Charles McClure and his team at Syracuse University;
	- discussion with key players. In addition to individual discussions (and we would repeat here our thanks to all those involved) the Study was discussed by the Advisory Committee on Performance Indicators of SCONUL;
	- an expert Workshop, which included international participation, and was informed by papers on UK work, on IFLA’s approach, on the new ISO standard and on the American work referred to above.

It was not the purpose of this Study to develop standard methodologies for the collection of underlying datasets. As with EAL this task would require further, quite extensive, work including testing within operational environments. This issue is referred to in the Recommendations on page 71.

The Study Team anticipate that it will be both desirable and necessary to subject this Report to further discussion: the Recommendations are intended to assist this process as well as identifying where further work is needed. It is hoped that this process may be accompanied by moves towards the definition and agreement of standard performance indicators within the international community. This is particularly important in the electronic library environment because libraries world-wide are accessing similar, and very often identical, resources.

Note: the symbol  is used in the remainder of this Report to indicate key new or revised performance indicators designed particularly for electronic libraries.

3.	Performance Measurement Model

In this Report we have focused on the library manager’s requirements for, and use of, performance indicators. However, it is important to bear in mind that there are other perspectives which can be taken. A stakeholder approach would also consider the needs of groups such as:

•	customers (students, academic staff, ......)
•	institutional managers
•	funding councils (including their quality audit teams)
•	government
•	other library managers (e.g. public libraries)
•	student advisors (e.g. careers advisors)
•	heads of academic departments
•	posterity (bearing in mind libraries’ responsibilities to future generations).

Although beyond the scope of this Report, it would be possible to identify overlapping sets of indicators for these groups. Generally, however, it is the manager who would need the most comprehensive approach. The stakeholder issue does, however, draw the attention of managers to the need to explore a wide variety of perspectives: it may be taken as axiomatic that the customer perspective s the most important of these, and that the precepts of quality management (‘fitness for the customer’s purpose’ and ‘conformance to the customer’s requirements’) provide the basis which underpins management decision making.

The basic model was used in this Study derives from the discussions held during the Scoping Study, and identifies MI/PM requirements on the basis of managers’ needs for information and data to inform their decision making.  Without suggesting that watertight compartments exist to separate types of decision-making it has been found useful to identify the following managerial tasks for these purposes:

		operational management, by which is meant the detailed day-to-day organisation and control of services and resources, including the management of exception conditions as they arise.

		forward planning, which includes the medium to long term planning of services and resource requirements and requires the extrapolation of current trends, an analysis of the external environment and the ability to carry out “what if” analyses.

		evaluation and review, which requires analyses of recent and current activity, with close reference to user requirements and user satisfaction, and increasingly looks to comparisons with similar service providers elsewhere (whether through formal benchmarking or more informally).

Clearly it will sometimes be the case that the same indicators can be used in more than one context and it may be the case that the same dataset can be used in a number of indicators.  In all cases it is important that reliable longitudinal datasets are built up to enable comparisons over time to be made.

A second dimension to the model is provided by the analysis of the electronic library in Section 1.1 above. The functions of the electronic library are described there as:

		resource discovery
		resource delivery
		resource utilisation
		infrastructure provision
		resource management

These library functions can be combined with the managerial ‘tasks’ to provide guidance on the sort of decisions which managers of the electronic library need to take (and hence need performance indicators to inform). Examples of the decision areas which managers will encounter are given in Fig. 2 below:

	Resource Discovery	Resource
Delivery	Resource
Utilisation	Infrastructure
Provision	Resource
Management	
Operational
Management	Ease of use of OPACs	Speed of document supply	Appropriateness of conversion software	Availability of PCs	Peaks in demand for services	
Forward
Planning	Which bibliographic indexes to provide?	Agreements for document supply	Emerging document formats	Number of networked PCs to be provided	Predicted costs of alternative services	
Strategic
Management	Clumps	Co-operative (e.g. regional) agreements	New classes of software tools	Development of off-campus infrastructure	IPR legislation for electronic sources	
Fig. 2: Illustrative Decision Areas for the Electronic Library Manager

In discussions with experts in the field during this Study one of the issues which was stressed repeatedly was that for the majority of decisions, and certainly at the forward planning and strategic levels, the emphasis needs to be on qualitative rather than quantitative approaches. The illustrative decision areas in Table 1 reinforce this point. In particular managers need to be concerned about the impact of their services (i.e. do the users actually gain benefits from them?) rather than their extent. This is particularly true in the electronic context where the temptation is to equate large numbers of accesses to, say, a web resource, with large benefit: this is not necessarily the case! There is a danger (which some approaches to library performance indicators have fallen into) of developing large numbers of quantitative performance indicators (‘measuring the measurable’) and losing sight of the key issues. We shall return to this later in the Report.

It is taken as an underlying principle that the number of datasets and indicators should be kept to a minimum.  Data collection is an expensive business and with the universally experienced pressure on resources only that data which can be justified by its subsequent usage should be collected.  It should always be remembered that data collection and manipulation is a “cost of quality”.

The sources of data for management information and performance measurement will be various.  In this Report, as in the Scoping Study, we suggest that it is useful to consider three categories of data source:

		data generated locally, as for example by monitoring use of a locally-mounted CD-ROM.

		data generated by local use of a remote data source, as for example in the use of a dataset mounted elsewhere and accessed across a network.

		data generated by other users and other libraries which is to be used for comparative (e.g. benchmarking) purposes.

As we noted above, one criticism of work in this area, and particularly of EAL, is that the number of indicators suggested is too large for it to be economic to implement in academic libraries, and too large for an overall conclusion to be drawn. There is an enormous amount of data potentially available and usable: the key issue is to find and use only the data and indicators which are both economic to compile and useful to management. For this reason we have highlighted what we regard as the key electronic indicators in our conclusions.

Finally in this Section it is pertinent to return to the issue of what performance indicators are for. As the term suggests they are indicators rather than measures. The prime purpose is to indicate to the manager:
•	whether the library’s performance has changed significantly (e.g. use of a particular database has suddenly accelerated);
•	whether the library’s performance differs significantly from that of other, similar libraries (e.g. costs of a particular service are much higher or lower);
•	whether the use of the library and of particular services differs significantly across different user communities (e.g. between disciplines);
•	and so on.

In each case there may be good reasons for such differences. Each university has a subtly different mission and in seeking to support the achievement of that mission each library will devise a different set of services, which will change over time in response to its users’ changing needs. Indicators should not be taken as absolute judgements of the ‘goodness’ of any particular library, but should inform a considered approach to management and decision making.

4.	Related work

In this Chapter we review the major contributions which have been made to the development of library performance indicators in recent years. This is not intended as an exhaustive summary of all the relevant work, but rather highlights studies and publications which have particular relevance to the MIEL Programme.

4.1 Van House, Nancy A., Weil, Beth T. and McClure, Charles E. Measuring academic library performance: a practical approach.  London and Chicago; American Library Association, 1990.  ISBN: 0-8389-0529-3.

Nancy Van House’s now seminal manual on academic library performance measurement builds on her work in 1987, when she revised Douglas Zweizig’s Output measures for public libraries: a manual of standardized procedures (ALA, 1987).  The purpose of this later work is to present a set of practical output measures for academic and research libraries, and thus measures have been designed which

•	evaluate the effectiveness of library activity
•	are useful for and replicable in all types and sizes of academic libraries
•	support decision-making
•	are easy to apply and use and are inexpensive to administer
•	are user-oriented
•	reflect common library goals and objectives

All the measures are service oriented and, as such, (and unlike some other works in this list) are not concerned with a library’s internal functions, such as speed of processing new acquisitions. The preface makes clear that technical services provide necessary support to public services, but the measures in this manual are concerned with the service as delivered to the user, not with the many intermediate processes within the library required to deliver those services.   Consequently, heavy emphasis is given to the design of library user satisfaction surveys to inform measures concerned with issues such as general satisfaction, reference satisfaction and online search evaluation.  There is a very scholarly account of questionnaire design, data collection and response analysis.  

In addition to the methodology of user surveys, the manual contains 15 measures in the categories of general user satisfaction, materials availability and use, facilities and library uses and information services.  Each measure is arranged according to the same format, comprising: definition, data collection, data analysis, discussion and further suggestions.

4.2 King Research Ltd. Keys to success: performance indicators for public libraries.  London; HMSO 1990.  ISBN: 0-11-430048-8.

This work was commissioned by the UK Government’s Department of National Heritage as part of a series of attempts to create and develop management tools for public libraries.  It was commissioned at a time of growing awareness of the importance of performance measurement in libraries, and the contract to produce the work was managed by the (then) British Library Research and Development Department.

A “how to” section at the beginning of the manual explains the rationale for library performance measurement, and how it may be achieved, to an uninformed readership.  The terminology of performance assessment is closely defined. A clear distinction is made, for example, between performance measures “indicators of size, ‘goodness’, use, usefulness, value etc.” and performance indicators “...derived from combinations of measures”.  Section 2 lists and describes the performance measures included in the manual; of which there are 21, divided into four groups:

•	service input cost measures
•	service output measures
•	service effectiveness measure
•	service domain measures

Section 3 lists 16 performance indicators which can be formed from the measures.  These also fall into four categories:

•	operational performance indicators
•	effectiveness indicators
•	cost-effectiveness indicators
•	impact indicators

The calculation and interpretation of each indicator is then described, in accordance with the following headings: measures, methods, correlation with performance, related indicators and examples.  The manual concludes with detailed methodologies for measuring input, output, effectiveness and domain values, and for calculating the constituent measures and the indicators derived from them.

4.3 Ward, S. et al.  Library performance indicators and library management tools. Luxembourg; European Commission DGXIII-E3, 1995 [EUR 16483 EN]. 

The PROLIB-PI study was commissioned by the European Commission in 1993 as part of the European Libraries Plan within the 3rd Framework Programme of DG-XIII of the Commission.  The aim of the study was to develop a toolbox of performance measure and indicators which are relevant to and applicable in all types of library within Europe.  The finished toolbox is the result of a review of trends and developments across the range of libraries in Europe regarding their use of management information and their application of performance measurement techniques.

The work was carried out during 1994 by a consortium comprising De Montfort University (Project Co-ordinator), the Library and Information Statistics Unit at Loughborough University and Essex County Libraries.

The toolbox which has been created provides guidelines on the implementation and analysis of a variety of measures and indicators useful in the evaluation of library services.  It takes into account the theoretical development of performance measurement techniques and decision support in library management.  Following widespread consultation with librarians and experts in the subjects, it also reflects the experience and practice of a variety of libraries in the data collection and analyses they undertake.

No specific management models have been developed for the toolbox because, De Montfort contend in their final report, “to a large extent, this is precluded by the differences in size and objectives of the individual libraries within each category.  Local circumstances dictate a local management style”.  The authors also maintain that their performance measurement techniques can be applied in any library, regardless of the management model.  The study provides a list of performance measures and concomitant indicators based on measurable and quantifiable aspects of a library service, that is: target population, provision (level/quantity), cost and use.  The study posits that insight into library performance can be gained by combining these measurable aspects to form ratios; for example, cost:provision, use:provision, cost:use, cost:target population and so on.  In addition, “timeliness”, “needs fill” and “user satisfaction” are included as measurable aspects of a library service, but as indicators in their own right.  Methods for data collection for all measures and indicators are described.

4.4 International Standards Organisation ISO 11620: Information and documentation - library performance indicators. Working draft 10. ISO Secretariat ISO/TC 46 SC8/WG4, 1995.

The International Standards Organisation established its Working Group on Library Performance Measurement (TC46/SC8/WG4) after a feasibility study conducted from December 1991 by Professor Charles McClure.  The remit of the Working Group was to compile a draft standard set of indicators for library performance measurement, and standardised methodologies for their compilation.  The design and publication of such a standard was intended to meet two desiderata which had been precipitated by the growing interest in performance measurement among international fora of LIS practitioners:

•	that the burgeoning number of library performance measurement manuals was leading practitioners to use widely-varying sets of indicators, and this was not held to be conducive to the establishment of best practice in an emergent and internationally important field

•	that, even when different manuals contained similarly-named indicators, their methodologies were often found to greatly at odds; and that this brought with it serious doubt as to the validity of the cross-comparison of values generated for similarly-named indicators in different libraries, with similar implications as above for the establishment of best practice.

The membership of the Working Party was principally drawn from IFLA members with a strong professional interest in the field.  Denmark, France, Germany, Norway, Russia, South Africa, Sweden, the United Kingdom and the United States of America were represented.

The brief for the standard was, and continues to be, that it:

•	is concerned with the evaluation of all types of library
•	shall specify the requirements of each performance indicator, including requisite constituent datasets and methodology
•	shall prescribe a core set of indicators
•	shall give guidance on how to implement performance indicators in libraries where such indicators are not already in use
•	shall provide a standardised terminology
•	shall contain concise descriptions of the indicators and of the collection and analysis of the data needed

In compiling the draft set of core indicators, the Working Group drew indicators which were either already in widespread use in libraries or which were well documented in the authoritative published sources then extant.  Because of this reliance on tried and tested indicators, none of the measures included to date address the need for management information in the electronic library; since thus far no such measures have yet found sufficiently wide general acceptance.  The standard has passed through several drafts, but the Working Group now hopes to make formal publication in 1998.  Thereafter, the Working Group will continue to monitor developments in best practice and to incorporate appropriate new indicators into subsequent editions of the standard.

4.5 IFLA.  Measuring quality: international guidelines for performance measurement in academic libraries. IFLA Section of University Libraries and Other General Research Libraries.  Saur, 1995.

The IFLA Section of University Libraries and Other General Research Libraries instituted a working group in 1990 for drawing up guidelines for performance measurement.  The main purpose of the guidelines, as commissioned by the Section, was to combine a set of performance indicators that would be applicable in academic libraries all over the world, and this brief naturally made it necessary to choose indicators that would fit a broad range of libraries.  The restriction to academic libraries only was solely as a consequence of the special mission of the commissioning section.

A first selection of possible indicators had been made at the workshop of the section in 1989, which identified five broadly supported candidates for inclusion.  Thereafter, the working group examined the existing literature on performance measurement, and discarded all indicators measuring only quantity, not quality of service.  It was decided to present only indicators that would allow for an immediate evaluation of the quality of a service and that could be set in relation to a distinct user-oriented goal of the library.  In order to give help for setting such goals, the handbook presents a general mission statement for an academic library including details about collection building, access and facilities, information policies, and preservation.  The handbook stresses the importance of defining the library’s mission and defining goals and objectives that are set in relation to the results of performance measurement.

From the outset, the working group had tried to keep the final set of indicators to a manageable size, as well as to try and ensure that they be user-oriented and easy to apply (in order to encourage their wider adoption).  Consequently the complete list now contains only 13 indicators, all documented in the literature and/or tested operationally by members of the working group or other libraries.  Each indicator is described with definition, aims, methods and possible reactions.  The working group has tried to show reasons for failure and to point out possible solutions and reactions based on the measurement.

A significant respect in which the IFLA list differs from some other collections of library performance indicators is that it contains no attempt to measure either cost-effectiveness or efficiency.  The working group has been aware that the measurement of cost-effectiveness is now more than ever a matter of great concern to library managers, but it opted not to propose appropriate indicators because of the great variation in the details of cost-accounting practices in use all over the world.  The group recognised that not only do sources and amounts of funding differ greatly in countries and nearly every library, but so do the details of terminology which are proper to the various unit costs.

The guidelines, as they now exist, concentrate on measuring the effectiveness of a library in meeting its goals; but a new project on cost-effectiveness is under consideration by the IFLA section which commissioned this work.

4.6 Commission of the European Communities, Fourth Framework (Telematics), Libraries Programme: Project EQLIPSE (Evaluation and Quality in Library Performance: System for Europe) and Concerted Action CAMILE (Concerted Action on Management Information for Libraries in Europe).

The EQLIPSE Project ran for two years from February 1995, and was co-ordinated by the University of Central Lancashire.  The project consortium comprised a total of ten partners from seven European countries and included academic, public, national and special libraries, a software development company and a library systems supplier.

The overall objective of EQLIPSE was to specify, develop and validate an open IT based system to support quality management and performance measurement in libraries of all types. The resultant system is based on client-server architecture and offers compatibility with library systems from various library suppliers.

The project operated through six phases, as follows:

1  Library Requirements analysis, in which a series of tasks brought together expertise from the ten partners and associates with internationally recognised quality management, library performance measurement and technological perspectives to define the libraries requirements from EQLIPSE and hence the groundwork for development.

2  Initial functional specification, in which the technical issues, including networking and systems integration, were reviewed, leading to the production of an initial functional specification.

3  Prototype System, in which the initial functional specification was used to design and build a prototype, functional system.

4  Data Tools and Data Collection, in which the partners adapted and designed tools for collecting data, supplementary to that contained in their operational IT-based systems, and built up test datasets for use with the system.

5  Field Trials and Evaluation, in which the prototype system was installed in two libraries, debugged and developed into a fully functional system.

	6  Integration in Libraries, in which the debugged system was fully trialed, in a pseudo operational environment, in six libraries (representing the different types of libraries found across the European Union, including those in Less Favoured Regions) and evaluated.

The consortium’s work resulted in the production of the prototype EQLIPSE system (phase 3) and in six public deliverable reports, presenting the works of phases 1 and 2, 4 to 6, and a Final Report (Deliverable 7).

The aim of the CAMILE Concerted Action is to promote and spread the results of the four European Commission Libraries Programme Projects included in the Action Line IV (Stimulation of a European market in telematic products and services specific for libraries) theme 18bis (Models and tools to support decision-making in libraries) funding line; namely EQLIPSE (see above), MINSTREL (Management Information Software Tool: Research in Libraries), DECIDE (Decision Support Models: a Decision Support System for European Academic and Public Libraries) and DECIMAL (Decision-making in Libraries: Decision Research for the Development of Integrated Library Systems). CAMILE began in August 1996 and has an expected duration of two years.

In order to achieve the above aim, the Concerted Action will develop and pilot a series of workshops for practitioners throughout the library community in Europe to disseminate the work of the four related EC projects. This will enable the exchange of information, ideas, and good practice between experts, and help them to consider and develop common approaches, identify areas for standardisation, and demonstrate the application of different software solutions. In particular, CAMILE will seek to:

	build consensus on the issues and activities relating to library and information management, including  performance measurement and the assessment of quality using both quantitative and qualitative techniques;

	provide a framework for technical and theoretical discussion, the formation of common policies and the development of standards;

	bring together activities and ideas at national, European and international levels, leading to the identification of areas requiring further research;

	provide a forum whereby information on common issues and challenges can be relayed back to the appropriate European Commission programmes;

	add value to the investment in Call for Proposals (CfP) 1993 projects through the sharing of research results and related technical discussion between experts;

	demonstrate decision-support technologies resulting from the four constituent projects.

The co-ordinating partners from each of the constituent CEC projects together form the Concerted Action Management Group, which will develop syntheses of their results and feed these into a series of thematic group discussions to form the building blocks for a model European travelling workshop.  The workshop will be run at a number of test sites contributed by the associated partners involved in the CEC projects, and individual countries' professional associations will be encouraged to continue the workshop series beyond the life of the separate projects.   

CAMILE is being co-ordinated by De Montfort University's Division of Learning Development (UK) on behalf of the co-ordinators of each of the projects.

4.7 McClure, Charles R. and Lopata, Cynthia L.  Assessing the academic networked environment: strategies and options. Coalition for Networked Information; 1996.

This contribution to the literature of performance measurement is the latest addition to the distinguished corpus of work in this area built up by Professor Charles McClure of Syracuse University, and has grown directly out of earlier publications concerned with the performance measurement of paper-based library services. The work results from a larger study, Assessing the academic networked environment, conducted from October 1994 to December 1995, which was set up to address the increasingly glaring anomaly that, although library services had by then attracted a mature literature of performance measurement, little work had thus far been done on the increasingly important area of networked electronic information sources.

Consequently, the manual represents an attempt to formulate equivalents, with respect to networked services, for the types of performance measurement questions which library managers have become used to ask about their paper-based services, and to postulate methodologies for answering them. It is important to note that the work addresses the whole academic networked environment, and not just the library element. The study posits the following questions as a foundation for assessing networked services:

•	What is the volume and type of networking taking place on a particular academic campus?
•	Who are the users that access the academic network and what types of services do they utilise?
•	How much do the various types of network activities and services cost?
•	How has access to and use of networked information resources and services affected teaching, research, learning, service, and other aspects of traditional academic life?

The study aims to equip managers with the tools to answer these types of question by achieving the following objectives:

•	to describe a range of techniques that assess the academic networked environment
•	to provide procedures for collecting and analysing the data needed to produce an assessment of the academic networked environment
•	identify and discuss data collection issues and problems that may be encountered when conducting such assessments
•	to encourage academic institutions to engage in a regular program of ongoing evaluation and assessment of their computing networks
•	to provide a baseline for conducting network assessments as a means for improving academic networked services.

In order to meet these objectives, the manual is divided into five parts, of which part 1 is an introductory overview.  Part 2 presents guidelines and suggestions for appropriate data collection techniques, and combines both quantitative and qualitative methodologies.  It may be noted that the document as a whole lays heavier stress on qualitative techniques than is usual in the literature of performance measurement, and the document may consequently be considered to offer library managers a richer choice of evaluation techniques than has previously been available. Part 3 presents the performance measures which the original study devised, as well as six key assessment areas in which quantitative data can be collected.  The measures themselves are each laid out in a standard form which includes definition, issues, data collection, data analysis discussion and additional suggestions, while the six areas suitable for qualitative assessment techniques include:

•	users: the number and types of users of the network and the frequency of their use
•	costs: the total and types of financial resources necessary to operate the network
•	network traffic: amounts and types
•	use: amounts and types
•	services: the applications which are available on the network
•	support: the types of assistance which are available to network users

Part 4 of the manual addresses the importance of user surveys in network assessment, and links back to the discussions of data collection methods earlier in the work.  Part 5 summarises the importance of ongoing assessment and suggests directions for further research.

Further work in this area by McClure et al. was reported verbally to the Expert Workshop for the current study, held in May 1997. In addition, the Coalition for Networked Information (CNI) launched a programme of field-testing of Assessing the Academic Networked Environment in March 1997, involving a range of institutions, including major institutions and community colleges. Interestingly, Kings College London is a member of the group.

4.8 East, Harry, Sheppard Elaine and Jeal, Yvette.  A huge leap forward: a quantitative and qualitative examination of the development of access to database services by British Universities, 1988-1994. London; British Library Board, 1995. [British Library R&D Report 6202].

This report was jointly commissioned by the Centre for Communication and Information Studies and the British Library Research and Innovation Centre, and examines the ways in which academic institutions incorporated externally-sourced electronic information resources, such as commercially produced databases into their service activities during the period studied.

The foreword makes clear that, at the beginning of the study period, the main mode of access to such services was by online searches conducted on files mounted on external commercial database hosts, and mediated for the end-user by library staff with appropriate expertise.  In fact, this form of access was then beginning a period of very steep decline, in which the external hosts have been replaced by the onsite installation of the required files on CD-ROM.  In fact, the authors quantify this effect very starkly: “By 1990 university libraries were, on average, already spending more on CD-ROM acquisitions than on fees to commercial hosts, and by 1994 in excess of ten times more”. The report acknowledges that one of the reasons for the ascendancy of CD-ROM as a delivery medium is that it permits a far wider degree of direct access for the end user than had previously been available, since a one-off or annual fee is sufficient to secure unlimited single user access.

The authors then go on to note that even the CD-ROM model is no longer valid since the advent of the BIDS-ISI service in 1991.  “By March 1995, 87 research and higher education institutions were subscribing to it, thereby affording their staff and students virtually unrestricted end-user access.  In terms of its rapid acceptance and growth in its use, this service had been judged by most of its users to be ’a huge leap forward’ in terms of accessibility”. The principle of ‘free at the point of use’ which characterises JISC Dataservices is highly significant in encouraging use and stimulating demand.

The report relate to aspects of this changing pattern of database access, and is presented in three parts:

•	Part 1 provides a summary of university expenditure on externally produced databases in a variety of media during the period 1988-1994
•	Part 2 reports on the series of studies undertaken by the research team, specifically in regard of the BIDS-ISI service, and with particular emphasis on who the users are, how they use the service, and how they perceive its advantages and shortcomings.
•	Part 3 is a selections of issue arising directly or indirectly from the studies in Part 2 and the authors’ observations on them. 

The observations in Part 3 are particularly concerned with the cost-benefit aspects of the changing model of externally-generated information use; and in this connection makes the point that the authors’ observations have shown that, though there is undoubtedly some sophisticated searching of these very powerful information retrieval tools, the majority of users continue to employ them in a fairly simplistic and uncritical manner.

4.9 HEFC(E)  The effective academic library: a framework for evaluating the performance of UK academic libraries: a consultative report to HEFC(E), SHEFC, HEFC(W) and DENI by the Joint Funding Council’s Ad Hoc Group on performance indicators for libraries. Bristol; HEFC(E), 1995.

The origins of this document lie with the recommendation, made by the Follett Report in December 1993, that a framework of coherent and generic performance indicators, suitable for assessing academic libraries, be established which would build on earlier work carried out by COPOL and SCONUL.  In response to this recommendation the Joint Funding Councils set up and Ad Hoc Group on Performance Measures for Libraries was set up and reported its findings in the form of this discussion document.  Since its publication a supplement on service standards and development targets has also been issued.

The indicators contained in the report have been arranged by the Ad Hoc Group into five areas:

•	Integration: the level of integration between the mission, aims and objectives of the institution and those of the library (5 indicators)
•	User satisfaction: surveys and other feedback (e.g. course review, suggestions) (5 indicators)
•	Delivery: are stated objectives being met and is the volume of outputs high? (7 indicators)
•	Efficiency: outputs related to resource input (9 indicators)
•	Economy: cost per student (7 indicators)

As the primary purpose of the work was stimulate debate and discussion, rather than to stand as a prescriptive list for the academic library community to follow, the discussion of each indicator contains no methodology for compilation.  This sets the work at variance with the other lists in the established canon discussed above, but the authors do provide a list of the constituent datasets which are required to form the indicators

The aims of the Ad Hoc group in taking this inclusive approach were to balance simplicity and representativeness, to encourage debate about why any specific indicator should be used, to provide a shared framework and measures for comparison (but in such a way that the framework could be tailored to local objectives and timescales) and to facilitate links with quality assessment and the audit approaches of any university seeking to use the indicators.

The first edition of the report invited responses from all interested professionals and, during 1995, a series of regional seminars was organised by the Ad Hoc Group to stimulate debate.  The revised version of the document, taking into account feedback from the target readership was published in 1996.  Thereafter, SCONUL recommended that the scope of the set of indicators be expanded to include assessment of networked services; as this area of activity had not been included in the original or revised editions.  SCONUL made funding available to support research towards the preparation of suitable indicators, and a researcher was appointed to the post thus created, based at the University of Cranfield, in April 1997.

4.10 Other Studies

A variety of studies of the use of network services have appeared recently. These probably represent the tip of the iceberg of individual studies carried out within institutions for their own purposes, although a few have wider significance. Reference has already been made to the annual reports of JISC Services. There is an interesting recent paper on web usage statistics which argues that counting web accesses is a fruitless task. In the United States McClure has been involved in a number of further exploratory studies, in addition to the work of McClure and Lopata referenced above., , , .

5.	Performance Indicators for the Electronic Library:
I Operational Management

5.1 Introduction
Operational managers need consistent and reliable sets of data about the services for which they are responsible. A process model approach may be particularly appropriate since at the operational level the key issue is to manage the use of resources (people, money, information, and so on) through processes to produce outputs (books lent, web pages accessed, etc.). 

Input		Process		Output	

To some extent operational management will also concern itself with outcomes - the effects of its outputs. However, in general indicators will be concerned with tracking the amount of service being provided per unit of resource (e.g. books issued per member of staff employed at the service desk). They will also facilitate ‘what if?’ comparisons to enable alternatives to be explored. The MAYHEM tool is a good example of a software product which enables such comparisons to be made to assist alternatives to be evaluated.

Increasingly, operational managers are putting in place Service Level Agreements (SLAs) which define the type of service and the extent of that service, both in relation to internal and external customers. It is to be expected that each SLA will require one or more performance indicators to be devised in order for both the provider and customer to monitor compliance.

In the electronic environment it is necessary to find new, reliable indicators of ‘service provided’. There is thus a basic issue here concerning the most appropriate measures to use: for example, use of electronic services could be measured by reference to connect time, number of sessions, number of concurrent sessions, number of hits, cost, number of active users or a variety of other factors. Care has to be taken with each of these possibilities, since it is possible that each could be affected by irrelevant and indeed uncontrollable variables. For example, connect time may well depend on network response times outside the control of the individual library, while number of hits could be drastically curtailed if a cache came into use.

We have noted that for most CD-ROM monitoring software, in most of the City University reports, and with most of the JISC services, the principal chosen measure is sessions (or a variant on it). This has a number of advantages:

	 it is time- and process-independent, in that it measures each occasion a user tries to do something (find information or whatever) rather than how (in)efficient they are at it or how they go about it, or whether the network infrastructure is efficient.

	 it enables sub-measures to be defined (e.g. by internal department)

	 it gives an (admittedly rough) idea of comparative use between services;

	 it gives the possibility of building up time series, although clearly only with great care;

	 it may facilitate inter-institutional comparisons, especially where the same service is provided to a number of institutions (as with JISC Services).

For those reasons this is our preferred approach to measuring electronic services. A formal definition of ‘sessions’ will be required, with examples, in order to implement indicators which use this concept.

5.2 Indicators for Operational Management
The actual indicators chosen by operational managers will be a matter for local decision, although - as we noted above - the acceptance of common measures, as when JISC services report the number of sessions with a particular dataset, will facilitate inter-institutional comparisons if used with care. Thus, for example, the lists of institutional total accesses in datacentre annual reports may raise questions for management about usage levels. This is referred to in the Recommendations on page 71.

It is useful for operational managers to consider their choice of indicators in relation to the five-fold functional model of the electronic library presented on page 3.

5.2.1	Resource Discovery
In this category we include bibliographic sources, other indexes and tools such as web crawlers,. There are two key issues here:

		the range of such resources, which defines the ‘map’ provided to users of the information landscape and therefore limits the landscape features to which they can gain access.

		the quality of such resources, equivalent to the accuracy and scale of the ‘map’, which defines whether items can be retrieved and, if they are retrieved, whether they are accurate, reliable, and so on.

We will consider further (in Chapter 7) the strategic significance of these issues for the provision of effective library services. For operational management we assume that the range of resources has been planned to meet as much demand as the resources available permit. The issue will be to manage the use of those resources. Therefore, the basic indicator for resource discovery services will be

	Sessions per service per month

Complemented by

	User satisfaction with service results

the latter enabling the manager to monitor whether the quality of each service is adequate. These indicators should form the basis of the performance indicators used for operational management of resource discovery.

In the ‘sessions’ indicator we have suggested using a time period of one month. The reason for this is that it accords with much accounting practice in UK universities (e.g. most institutions use a monthly financial reporting period, staff salaries are paid monthly, JISC dataservices provide monthly reports, and so on). However, there is no necessity to use this period if another is more suited to a particular site. Undoubtedly there will be a need to use similar indicators over different periods: for example, a manager may wish to examine peaks and troughs in demand for a particular service, so that counts of ‘hits’ over very short periods (e.g. each minute over a day) would be appropriate. Conversely satisfaction may be measured over much longer periods. This is a matter for local decision.

5.2.2	Resource Delivery
The indicators used for resource delivery will again depend, of course, on the services offered. As in resource discovery (and recognising the overlap between the concepts) the ‘sessions per service’ and ‘user satisfaction’ indicators will be central. Each service (i.e. each external dataset to which access is provided, each internal dataset including CD-ROMs) will thus be defined in a similar manner. Where it is possible to do so, it may be desirable to maintain a parallel indicator, as follows:

	Items downloaded per service per month

Some services, such as web pages, are not amenable to this type of analysis (see page 35). There are also severe problems in defining the concept of an ‘item’ or ‘document’ in the context of electronic resources: we discuss this further on page 58. Any attempt to measure activity of such information services must be taken with extreme care, especially if a time series is being compiled. As an alternative, for services where ‘items’ cannot be defined and ‘sessions’ is inappropriate or cannot be measured, it may be necessary to use:

	Number of ‘hits’ per service per month

where ‘service’ could include objects such as web pages. It may be noted that McClure and Lopata suggest this type of measure as a means of indicating use of network applications (e.g. word processing packages) over time. This is a particularly appropriate indicator for ‘front end’ services, such as web pages provided by the library and giving links to high quality web resources, where the effects of caching etc. are likely to be less pronounced (although again they cannot be ruled out).

5.2.3	Resource Utilisation
Under this heading managers may wish to have information on the availability of tools which users need to exploit resources (such as personal bibliographic software) and the extent of use of those tools. Therefore it may be helpful to define two indicators:

		User satisfaction with resource utilisation tools

		Percentage of users using each tool

The first indicator will provide information on whether the right range of tools is available: the second on the utilisation (or market penetration) of each.

5.2.4	Infrastructure Provision

Operational managers will need information on the adequacy and use of the infrastructure provided. This will include whether sufficient workstations are available, whether the network is adequate and reliable, whether support services are available and adequate and so on. The type of measures required will be:

	Queuing times for access to workstations

	Downtime (as % of total time) per month

	Availability (as % of attempted accesses) per month

‘Downtime’ can be defined as the amount of time that the service is not available to users, either through planned hardware/software maintenance or through a system/network crash. ‘Availability’ is slightly different, as it measures ‘immediate access’, which may be denied where the library subscribes on the basis of a limited number of concurrent users, even though the physical connection is available.

Another useful measure will be to assess output generated by monitoring print, still a favourite form of output with most users:

	Pages of print per month

One of the key services for the electronic library is the provision of some kind of ‘help desk’ service. In general it will be necessary to monitor this through the collection of data on amount of activity and on user satisfaction. The latter may be measured by a negative, ‘number of complaints’ received, although this will not generally be satisfactory. Reference should be made to section 7 below for further comments on the measurement of satisfaction. However, for operational service management we suggest two indicators:

	Number of enquiries received per day

	User satisfaction

Note that a short timescale is suggested for the activity indicator.

5.2.5	Resource Management
The indicators suggested above will of course be used by managers to manage resources. In addition, many managers will also wish to explore how usage is spread across their user population. For example, they may wish to examine usage sub-divided by user status (student, staff, etc.) or by subject (Department, School, Faculty) or by site (including on- and off-campus) or by originating workstation or by any combination. They may also wish to identify non-use (perhaps as a prelude to an awareness-raising campaign). Generally it will be best to limit the number of such analyses to ensure that the quantity of data which has to be processed remains manageable.

For all operational indicators it will be useful to introduce an indicator of efficiency by using the cost of providing the service (or a surrogate in some cases, such as staff time) as a denominator. Hence a series of indicators will be used, such as 

	Number of sessions on each service/subscription cost

	Number of helpdesk enquiries per staff day

5.3 Issues
Managers need to be aware of a number of issues which could affect the validity of the data they collect for these purposes, and which may influence their data collection and analysis. These issues include:

	 much monitoring software will not be able to distinguish who is using a service, so that sub-division may be impossible or misleading

	 costs are notoriously difficult to assign to individual services. For example, what proportion of the (capital + revenue) costs of a fileserver should be assigned to a particular application which shares it? A similar argument applies to staffing hours and costs.

	 helpdesk activity may be related to matters outside the manager’s control (e.g. the user who imports a virus onto his/her PC despite warnings and then is dissatisfied with the helpdesk response). An SLA, with associated performance indicators, may offer an appropriate way of focusing attention on the service which is actually offered. 

	 activity indicators will be affected by factors outside the control of the user, the library, and the institution. For example, poor network response times will influence take-up of even the best service, but the root cause could lie at any one or more points between the user’s workstation and the service provider.

	 low usage does not mean low value. It is a matter for institutional decision as to the level of usage which is needed to justify provision of a service.

	 ‘hits’ can be a very poor indicator of value, since web pages (for example) may be accessed to discover if they are of value rather than as a result of accessing higher level metadata, and they may contain poor quality information which appears to be adequate to many users.

	 the effects of internal caching may need to be discounted.

5.7 Validation of Indicators

Clearly there is a need to validate the above processes.  They contain many defects but appear to the study team to offer the closest approximation available to us at present.

In order to test out this methodology it will be necessary to undertake testing in a small number of institutions.  In particular we need to check:

	-	To what extent can we identify and count electronic documents delivered by particular services?

	-	Do these indicators behave, over time, in a way that aids decision making?

	-	How sensitive are the indicators to variations in the delivery environment?

As with all the indicators suggested in this Report, we recommend that studies of their practical application to electronic library services are carried out.

6.	Performance Indicators for the Electronic Library:
II Forward Planning

Forward planning will encompass a considerable amount of ‘intelligence’ gathering concerning the environment in which the library operates (both in institutional and broader sectoral and societal terms) and likely developments which will have an impact on the library service.  In respect of the electronic library the rate of change will make qualitative measures, and especially assessment of such issues as relevance, currency, etc., vital. It will be important that the manager of the electronic library maintains a ‘technology watch’ function to predict the impact of new technologies and new products on the services offered.

To an extent forward planning will also depend on the extrapolation of current trends, and it is in this area that hard management data may be of most use.  Typical data in this category might be:

		changes in the size and composition of the market
		extrapolation of market penetration
		extrapolation of individual service usage

For example, the manager may wish to develop a set of time series of the type shown below (using year 0 as the current year):

	Size of market
		Number of students...... in years -2, -1, 0, 1, 2, 3 .......
		Number of staff...... in years -2, -1, 0, 1, 2, 3 .......

	Market penetration
	Proportion of students as active users in years -2, -1, 0, 1, 2, 3 .......
	Proportion of staff as active users in years -2, -1, 0, 1, 2, 3 .......

	Resources
	Available budget for services in years  -2, -1, 0, 1, 2, 3 .......
	Available budget for staff in years  -2, -1, 0, 1, 2, 3 .......

	Use of services
	Total number of sessions in years  -2, -1, 0, 1, 2, 3 .......
	Number of sessions per service type in years  -2, -1, 0, 1, 2, 3 .......

Cross-sectoral data will be important for forward planning, both to enable comparisons with the sector as a whole to be made and to point up particular issues which may indicate that the local service will need to consider a new approach or new service. More generally benchmarking approaches, using data from the local and comparative libraries (or a carefully constructed group of comparator libraries) will be important. Again it is worth stressing that performance indicators must be used to indicate issues that the manager needs to address, not to build up spurious league tables which then form the manager’s real agenda.

7.	Performance Indicators for the Electronic Library:
III Evaluation and Review

In this Section we are particularly concerned to build on the approach taken in The Effective Academic Library, which uses a fivefold structure to gauge overall library effectiveness, as shown below:

						   P1	Integration
						   P2  	Quality of Service
Overall Library Effectiveness   =		   P3 	Delivery
						   P4	Efficiency
						   P5	Economy

McClure has suggested that a similar set of criteria might be appropriate for the electronic networked environment, as follows:

	 Extensiveness
	 Efficiency
	 Effectiveness
	 Service Quality
	 Impact
	 Usefulness
	 Adoption

It would be useful to carry out a detailed comparison of the EAL and McClure categories. However in this Chapter we will look at each of the performance indicators presented in each of the EAL categories, referencing other approaches as appropriate, and suggest how they can be adapted and supplemented to meet the needs of the electronic library.  We make the assumption that for the foreseeable future libraries will operate a hybrid ‘traditional and electronic’ service and that the performance indicators will need to reflect that diversity.  Thus the suggestions here represent an adaptation, not a replacement, of EAL’s proposals.

It is noted that work is continuing to refine the EAL approach and to identify a more compact set of key performance indicators which could be used by academic libraries and by external bodies to inform judgements on effectiveness and efficiency.

7.1	Integration
In this area the needs of the electronic library and those of the traditional library are very similar,  The key issue of integration is “identifying the level of integration between the mission, aims and objectives of the institution and those of its library service”.  The indicators which are identified are as follows:

P1.1	The cohesiveness between the mission, aims, objectives and strategic plan of the institution and those of its library
P1.2	The resourcing mechanisms used by the institution to provide its library service
P1.3	The academic and research planning processes and outcomes
P1.4	Liaison between service providers and users
P1.5	Internal assessment and audit mechanisms

7.1.1	Performance Indicator EAL 1.1: The cohesiveness between the mission, aims, objectives and strategic plan of the institution and those of its library

EAL states that “evidence of a clear information strategy for the institution, linked with a clear component for the library is expected.

Most institutions now have information strategies in place.  While it is difficult to draw firm conclusions, there is some evidence that strategies lean heavily towards the place and development of IT in institutions.  For the electronic library, the indicator needs to demonstrate how the IT infrastructure (actual and planned) enables information service delivery to be planned and achieved and that the planning of electronic library services is an integrated part of the institutional strategy. The issue of effective delivery of services to the desktop will be a crucial issue for consideration. For example, how is the library’s planning linked to the provision of workstations for academic staff and students?

7.1.2	Performance Indicator EAL 1.2: The resourcing mechanisms used by the institution to provide its library service

The electronic library issues are the same as those for the traditional library, although it may be necessary to examine the extent to which resources are distributed e.g. for the provision of infrastructure including PCs.

7.1.3	Performance Indicator EAL 1.3: The academic and research planning processes and outcomes

The electronic library issues are largely covered by this indicator, although evidence of planned co-operative provision may need a different emphasis.

7.1.4	Performance indicator EAL 1.4: Liaison between service providers and users

Again, the electronic library issues are largely the same as those described but there is scope for the use of electronic surveys and suggestion boxes, and these may be particularly important for off-campus (or all out-of-library) delivery.

7.1.5	Performance Indicator EAL 1.5: Internal assessment and audit mechanisms

The issues here are the same as those for the traditional library, except that consideration may be given to sector-wide issues such as reviews of the effectiveness of services (such as JISC datasets) provided for the community as a whole.

It is perhaps worth adding a note here to point to the increasing concern about the extent of non-use of electronic services.  A particular integration issue will be the proportion of users who do not access services, the reasons for non-use and the integration of non-use issues into institutional planning processes.  The same issue will also need to be picked up elsewhere in the evaluation and review process.

7.2	User Satisfaction

Although the overall issue of user satisfaction is the same in traditional or electronic library services, there will be significant difference in the detailed data collection and indeed in the questions which need to be asked.  It is noted in EAL that the assessment of user satisfaction needs to include both formative and summative aspects, and again this is true of the electronic library.

EAL suggest the use of five user satisfaction performance indicators, as follows, but does not define them further.

			P2.1	Overall user satisfaction
			P2.2	Document delivery services
			P2.3	Information services
			P2.4	Study facilities
			P2.5	Information skills programme

For the electronic library, it is suggested that the following additional user satisfaction performance indicator may be required.

		P2.6	IT infrastructure

It is recognised that this may be a difficult area since such an indicator is highly relevant to converged services but may not fall within the remit of non-converged libraries.  Nevertheless, electronic library service effectiveness cannot be judged without it.

We would suggest that the assessment of user satisfaction with the IT infrastructure needs to incorporate:

		satisfaction with end-user equipment (e.g. is a PC available and if so is it of adequate specification?)
		satisfaction with end-user software (e.g. is Adobe Acrobat available to enable .pdf files to be manipulated?)
		satisfaction with network performance (e.g. availability, response times)

In relation to the first five indicators, assessment will need to include the following issues.

7.2.1	Performance Indicator EAL P2.1: Overall user satisfaction

Satisfaction with services made available to users but not necessarily through the library e.g. end-user access to JISC data services.  It will be important to identify the causes of dissatisfaction and the remaining indicators should be designed with that in mind.

7.2.2	Performance Indicator EAL P2.2: Document delivery services

Since electronic delivery will be a major focus of the electronic library it is important that this indicator reflects both traditional and electronic sources.  Indeed it may be particularly important to differentiate user satisfaction between the two, especially during the next few years as service development becomes more and more rapid.  Differences in user satisfaction between electronic and print will be significant issues for management.

7.2.3	Performance Indicator EAL P2.3: Information services

Information services will again form a major component of the electronic library service portfolio, so the remarks made under 7.2.2 should be repeated here.

As helpdesk services tend to be central to electronic services it may be appropriate to include a specific customer satisfaction indicator on this service (see also section 5 above on the operational management issues). McClure and Lopata suggest measuring:

	 volume of requests
	 type of requests
	 response time
	 accuracy of response
	 courtesy of staff

We would suggest that this comprehensive set is collected on a sample basis, so supplementing the data collected for operational management purposes.

7.2.4	Performance Indicator EAL P2.4: Study facilities

Assessment of customer satisfaction with study facilities will need to be expanded to include:

		in-library facilities where IT equipment is provided (range of facilities, number, suitability, availability etc.)
		on-campus facilities outside the Library from which sources can be accessed
		off-campus facilities if available

There is an interesting question as to the extent to which “study facilities” such as student study-bedrooms, work-based or even home-based facilities could or should be assessed.  In the context of lifelong learning these facilities could be crucial: they offer a good example of elements of the electronic library which may be largely or wholly outside the manager’s control but which will affect fundamentally the success of the service.

7.2.5	Performance Indicator EAL P2.5: Information Skills Programme

This indicator will need to encompass

	 induction in the IT facilities provided by the University
	 induction in IT skills (e.g. keyboard, mouse, log-on, Windows)
	 training in specific IT skills (e.g. use of packages/services including Web)

		as well as the information content issues in their electronic context.

Although discussion on the measurement of user satisfaction continues in a number of contexts it may be that as the library expands into electronic services there should be a closer look at some of the more general service sector customer satisfaction techniques which are available.  For example, the SERVQUAL methodology is based on the idea of assessing the gap between users’ expectations of a service and their perceptions of the service.  The five RATER criteria used in this analysis are:

		 Reliability
		 Assurance
		 Tangibles
		 Empathy
		 Responsiveness

(A sixth issue - Accuracy is probably also required to avoid the situation where customers are satisfied with a well-packaged but poor product!).  Although criticised in some quarters this type of methodology emphasises issues which are of importance to service users as well as indicating where managerial action is needed to improve customer satisfaction.

7.3	Delivery

Since EAL characterises this set of performance indicators as the “service outputs” they will clearly be fundamentally affected by electronic library developments.  Note also the connection with section 5 above and the micro-indicators of outputs needed to assist operational management.

The EAL performance indicators in this section are:

	 P3.1 Meeting service standards over a given period (local measure)
	 P3.2 Meeting development targets over a given period (local measure)
	 P3.3 Documents delivered per FTE student during a year (national measure)
	 P3.4 Enquiries answered per FTE student during a year (national measure)
	 P3.5 Proportion of students receiving post-induction instruction in information-handling skills during a year (national measure)
	 P3.6 Library study hours per FTE student during a year (national measure)
	 P3.7 Volumes in collection per FTE student at a given date (national measure)

It is important to emphasise that the first two are “local measures” while the remaining five are described as “national measures” and thus available for comparison/benchmarking purposes.  As discussed below, this second purpose is fraught with potential and real difficulties in the electronic library environment.

7.3.1	Performance indicator EAL 3.1: Meeting service standards over a given period (local measure)

Since this service standards are locally determined it is difficult to be specific about their content.  EAL provides an Annex containing examples of service standards.  If we extrapolate from these into the electronic environment, and then take into account the perspectives of electronic information service providers (such as commercial online services, JISC datacentres, etc.) then local service standards might be extended by the addition of the following:

Electronic database services

			To meet published service availability times

	(a) 	Service available to individual user without queuing 99% of 24 hour day/365 day year.
	(b)	Downtime advertised 24 hours in advance: 100% 

			To meet local ‘helpdesk’ availability targets

		(a)	Telephone support available during all advertised times
		(b)	Email support available 24 hours day/365 day year
		(c)	Telephone answered within 10 rings, 95% of day
		(d)	Email response within 24 hours, working days only
		(e)	Problems fully resolved: 95% within 48 hours

Many other examples of appropriate measures could be given in the local context.

7.3.2	Performance Indicator EAL 3.2: Meeting development targets over a given period (local measure)

Electronic library development issues will need to be included in planning and progress evaluated.  In practice this should not affect the performance indicator definition, except to note that service delivery in the electronic library tends to be more dependent on third parties (e.g. national service providers, local infrastructure providers) and this will need to be taken into account.

7.3.3	Performance Indicator EAL 3.3: Documents delivered per FTE student during a year (national measure)

EAL makes reference here to the need to include “electronic documents delivered” in the basket of items defined for third performance indicator.  The definition of “electronic documents delivered” is difficult, yet crucial to the successful management of the electronic library.  For that reason we give it extended consideration here.

7.3.3.1	Electronic documents

The difficulties in defining what we mean by an “electronic document” encapsulate the problem of achieving a satisfactory definition of the numbers delivered.  Electronic ‘documents’ will include:

			Web pages (or ‘sets’ of linked pages?)
			Journal papers published in electronic form
			Images contained within an electronic journal paper
			Scanned texts, of many kinds, derived from paper documents 
			Electronic books
			CD-ROMs
			Short electronic texts (of the type provided by newsfeed services)
			Digitised maps
			Digital video
			Digitised images
			Chemical formula data
			Numeric data streams (e.g. from satellite observations)
			........................ and so on.

7.3.3.2	Library responsibility

It is pertinent to ask to what extent the library can or should take responsibility or credit for document delivery which by-passes the library itself.  For example, once users have access to a web browser they will undoubtedly surf the net and download many thousands of web pages.  The library may have provided the browser, may have undertaken training, and may offer support on demand - but may have contributed little else.  Should such usage be counted in the ‘basket’ of documents delivered?

On the other hand many academic libraries have designed and provided web pages to provide links to ‘quality assured’ servers (and the same is true nationally of eLib projects like SOSIG and EEVL).  This is akin to the Library providing a printed guide to the literature followed up with an efficient interlibrary loan service.  As such it surely needs to be measured in some way.   We define such pages as ‘front-end’ services since they are akin to a user interface proving access to a range of resources.

It may also be necessary for institutions to consider web pages provided by academic staff (i.e. internal publishing) although such resources have not been covered in this Report since they would not normally be a library responsibility.

7.3.3.3	Electronic Documents and Effectiveness
In an earlier draft of this Report, which was considered  and commented on by a number of experts, we attempted to define document delivery measures based on accesses to web pages, email and other services. Rightly, these approaches engendered a great deal of debate. After much deliberation, we have removed these tentative indicators from the final version of this report, and taken a different approach.

The argument for this is as follows. Firstly, we recommend the retention of measures of usage of individual services as an operational management issue (see Chapter 5 above), thus recognising that for the management of electronic libraries such data is important. However, this Chapter is concerned with effectiveness. The number of electronic documents downloaded (even assuming that we have managed to resolve the definitional problem of what an electronic document is) cannot be regarded as an indicator of effectiveness. Unlike, for example, a traditional interlibrary loans service, where it is reasonable to assume that the majority of items delivered will have some value to the user, this may not be the case in the electronic environment. For example, web pages are accessed to determine their relevance (they currently act as their own metadata), are accessed accidentally, or are used as a route to other information. Furthermore, it rarely matters to the library how many items are accessed since most services are now provided under unlimited access contracts and released to users as ‘free at the point of use’. The same argument applies even where licence restrictions allow access only to a specified number of concurrent users.

What is important to the user is the range of resources available and their depth. These issues should therefore form the focus of effectiveness indicators. They are equivalent to indicators of the range and depth of traditional library collections and are considered under section 7.3.7 below.

7.3.4	Performance Indicator P3-4: Enquiries answered per FTE student during a year (national measure)
	
Provided that the definition of ‘enquiries answered’ is taken to include methods such as email used to provide the enquiry service then the electronic library should not pose particular problems in computing this indicator.

It is noted that this is one of the EAL indicators criticised on the grounds that it is unclear whether libraries should aim for a high or low score.  This problem is exacerbated in the electronic library environment and is especially problematic for converged services where a high level of enquiries may be generated by academic staff, rather than library staff, activity.

7.3.5 Performance Indicators P3-5: Proportion of students receiving post-induction instruction in information-handling skills during a year (national measure)

The main difficulty with this indicator for the electronic library is the need to relate it to intra-institutional responsibilities.  If IT training, and in particular post-induction IT training, is not a library responsibility then the indicator makes little sense for the library manager.  At institutional level the issue becomes one of whether IT training is embedded in the curriculum or is a ‘service’ responsibility. It may be that the indicator needs to be related closely to those designed in the first category, “Integration”.

7.3.6	Performance Indicator P3-6: Library study hours per FTE student during a year (national measure)

Again there is an issue here between converged and non-converged services.  We suggest that an additional indicator is adopted

		P3.6A   PC hours used per annum divided by FTE students

		where PC hours is defined as the number of PCs from which students can gain access to electronic library services multiplied by the number of hours they are used per annum, determined through sampling (automated where possible).

7.3.7	Performance Indicator P3-7: Volumes in collection per FTE student at a given date (national measure)

There is no obvious equivalent to ‘stock’ for the electronic library since the aim will be to enable access to as wide a proportion of the global information landscape as possible.  What is needed is an indicator of the range and depth of the electronic sources which are made available.  

In theory a Conspectus style approach would be rewarding, in which the library’s access arrangements were mapped against an authoritative list of possible information sources and services. AT present we lack such a list, and indeed in a fast-moving world it would not be feasible to produce one. The nearest surrogate available to UK academic libraries might be the full set of JISC dataservices: these have been selected by expert panels for their relevance and are potentially available to all institutions.

We therefore suggest a new indicator

	P3-7A	Proportion of JISC datasets available to users

Of course there remains an issue about other external datasets and about locally-mounted datasets.  We therefore suggest a second indicator. 

	P3-7B Total major electronic subscriptions

which will be the total numbers of

	JISC dataservice subscriptions
	Networked CD-ROMs (or equivalent)
	Commercial service subscriptions (above an agreed threshold value)

It should not be assumed that a low figure for either of these indicators equates with a poor service: as with other indicators it merely raises a question for management to consider.

7.4	Efficiency

Following the EAL approach, we do not attempt to define the cost of each element of the service but rather to build on the performance indicators provided to provide an overall view.  However in this area we take the view that most of the EAL indicators should be supplemented rather than modified to encompass the needs of the electronic library.  We note again the difficulties posed by converged services, especially for the definition of “library staff”.

7.4.1.	Efficiency of document delivery

		P4.3	Documents delivered/FTE library staff numbers
		P4.4	Total library expenditure/Document delivered

Because of the view taken over electronic document delivery in section 7.3, these indicators are not relevant to the measurement of the electronic library’s performance.

7.4.2	Efficiency of enquiry services

		P4.5	Enquiries answered/FTE library staff numbers
		P4.6	Total library expenditure/enquiries answered

Following P3.4, these indicators can be adapted to include electronic library considerations.

7.4.3	Efficiency of IT use

		P4.7A	Total library expenditure/PC hours used per annum

A new indicator following on from P3-6A (section 7.3.6 as above).

7.4.4	Efficiency of access

Indicators P4.8 and P4.9 relate to the efficiency of collection provision.  It would seem logical to try to provide an efficiency indicator related to access, although it is not immediately apparent how this could be achieved.  The following approach is suggested:

		discount access to free services, such as most web sites.
		discount non-networked CD-ROMs (on the basis that they have limited access: also on the impossibility of checking eligibility given the number of books, journals etc. which are accompanied by free CD-ROMs).
		count networked CD-ROM titles (not disks) PLUS JISC services subscriptions PLUS commercial service subscriptions where the annual fee paid is above an agreed threshold (as in P7.3B above).

Adopt new indicators:

		P4.8A	Total major subscriptions/FTE staff numbers

		P4.9A	Total expenditure/Total major subscriptions

Further work is needed to refine the “basket” which comprises ‘Total major subscriptions’, and to check the workability of the proposed indicators. (See Recommendations on page 71.)

7.5	Economy

Many of the issues under this heading are similar to those under 7.4 above.  Again a mix of modification to existing indicators and new indicators represents the best way forward.  Thus P5.1 and P5.2 may be taken as defined in EAL, provided expenditure on electronic services is included.

P5.3 is of little relevance to the electronic library, but following P3.6A, a new indicator should be introduced:

	P5.3A	PC hours available per annum per FTE student

P5.4 becomes less and less relevant in the context of the electronic library.  This is a difficult area: it could be argued that a library serving a dispersed population (e.g. a high proportion of off-campus students) through electronic services will incur additional costs (e.g. through the network infrastructure, helpdesks etc.) although the evidence of this is not yet available.  It is suggested that more research is needed in this area to establish the comparative costs of providing on and off-campus services in the electronic environment (or perhaps in and ex-library services).  An indicator with dispersal of delivery as the denominator might then be devised.

P5.5 needs to include electronic sources (as EAL states).  P5.6 is acceptable as it stands.

P5.7 should be extended by an additional indicator.

	P5.7A 	FTE students per network PC

Earlier notes on converged/non-converged services are relevant here.

7.6	Conclusions

The suggestions in 7.1 to 7.5 above would see the modification of a number of EAL indicators to include electronic library elements, and the insertion of a small number of additional indicators.

It is useful also to consider whether there is a need for any further indicators concerning electronic library services for particular purposes. The most obvious area for further work would be in impact studies, which might lead to ways in which reliable indicators of the impact of library services on their users could be calculated. The British Library Research & Innovation Centre is initiating work in this area.

An equally pressing issue is whether the number of indicators is too large for libraries to implement. As noted earlier in this Report, there is ongoing work to identify the key indicators from EAL. 

Appendix 2 provides a complete list of indicators.

8. Conclusions

The development of robust management information and performance measurement systems for electronic libraries is likely to require concerted effort over a considerable period of time. This Report provides a framework on which further developments can be built and also identifies a series of issues which need to be addressed. These include:

•	The need for a clearer definition and understanding of what is meant by the term “electronic library”. Since the eLib programme was set up, our understanding has developed in several directions. For example, convergence (of academic libraries and computing services) has continued apace and is now an accepted feature of a significant proportion of institutions, leading to questioning of what the term ‘library’ now means; the ‘hybrid library’ concept has emerged, reinforcing the need to consider electronic services alongside (and not ‘instead of’, or ‘as well as’, or ‘as a development of’) traditional services; the term ‘clumping’ has entered the vocabulary, with its emphasis on user access to multi-agency services; the concepts of ‘resource discovery’ and ‘resource delivery’ in the context of a global information network have become widespread; and so on. What then is the electronic library whose activities we are seeking to measure?
•	Alongside this issue, the present study has highlighted the problem of boundary definitions in academic institutions. When the library is no longer delineated as a physical space and, more fundamentally, when student-centred and problem-solving approaches to learning move information access and usage centre-stage in educational delivery, how do we define what is “library”? So, for example, is the provision of access to web pages a “library” or an “institutional” responsibility? Even to pose such a question points up the absurdity of rigid boundaries. Yet, this being so, what exactly is being managed by the electronic library manager?
•	Concern has been expressed over the years about the tendency of library performance measurement to become over-elaborate, to concentrate on ‘measuring the measurable’ and as a result to miss the big picture. It can all too easily become a matter of counting the trees rather than monitoring the growth of the forest. SCONUL’s current work on refining the EAL indicators is, in part, a response to this problem. In this Report we have focused on the development of a small number of key indicators for the electronic library, but more work is needed to ensure that these are the most appropriate set, that they are robust and that they meet the need. More than that we have concluded that more work is needed on the broader issue of impact, which at the end of the day is the justification for the existence of services.
•	Taking the issue of impact a stage further leads to the question of service quality. If we use the classic definitions of fitness for purpose and conformance to requirements then there is clearly a need for more work to elucidate these issues in the electronic library context. Although user satisfaction measures are important, they need to be supplemented. We have suggested the use of methodologies such as SERVQUAL as well as stakeholder approaches in this context.
•	After considerable investigation and discussion we arrived at the use of sessions as our basic unit of measurement for electronic services (see page 37 for a discussion on this). Our expert workshop agreed with this finding. However, this unit has several imperfections. To give but two examples: numbers of sessions will tend to decrease as front-end software becomes more sophisticated, allowing users to search across multiple datasets, as for example when a clump is used instead of a single catalogue; likewise sessions may be meaningless if ‘push’ technologies become widespread. Again, more work is needed in this area.
•	There is a raft of issues arising from the convergence of information and communications technologies. The most obvious are exemplified by web access, where the effects of caching and mirroring may subvert attempts at measurement. Email is often used to deliver information, but is it sensible to even try to disentangle the use of email for this purpose from its use as a communications medium? To do so would be akin to opening all members of an institution’s snailmail (personal letters, invitations, junk mail, and so on) in order to count interlibrary loan transactions!
•	EAL suggested that efforts should be made to incorporate a measure of electronic document delivery in the ‘basket’ of measures used to produce the overall document delivery indicator. We attempted to do this. However, our conclusion has been that such a measure would be entirely artificial, would serve no useful purpose and would be likely to detract from the meaning of the current indicator. While recognising that a service which delivers documents electronically rather than on paper may thus appear to be undertaking fewer transactions than its traditional neighbour as judged by this indicator we do not believe that a spurious measure is justified. Trying to measure the number of documents delivered detracts from the key question of the range and depth of services to which access has been made available. Our recommendation is that it is this latter approach which should be pursued.

9. Recommendations

Arising from this Study a number of important issues have been identified which require further attention. In this final Chapter we have highlighted what we believe are the most important matters for further attention.

9.1 Testing of the proposed indicators

Work is needed, involving a range of institutional contexts (‘old’ and ‘new’ universities; geographical spread; converged/ non-converged, etc.), to test the suggested indicators in live operational environments. See also section 9.5 below.

9.2 Agreement on the core set of indicators

It would be inappropriate to seek community agreement on a ‘core’ set of indicators until testing had proved the robustness and meaningfulness of the developed indicators in a live setting. However, once that process has been achieved, there will need to be a process whereby the set of indicators can be agreed. It would be appropriate for this process to be integrated with that used for EAL and its core set.

9.3 Modelling the electronic library

One of the difficulties of the present exercise has been the inadequacy of current models of the electronic library. JISC’s current calls for proposals in the areas of clumps and hybrid libraries are evidence of work which is being undertaken to develop our understanding of the organisational presence and reach of the electronic library (see also below). Alongside these developments there is a need to explore the management information needs of, and performance indicators for, emerging models of the electronic libraries. It is important, therefore, that work continues alongside these broader developments.

A further issue which may be significant is the development of distributed library services, where the emphasis is on off-campus delivery. Current emphases on lifelong learning, the new Government’s intention to establish a ‘University for Industry’ and the outcomes of the Dearing Review of Higher Education, may all shift demand away from traditional on-campus users towards a more fragmented and dispersed user population. Drawing on the recommendations of the Dearing Review in particular, there is a strong case for a study of management information needs and strategies in a variety of electronic and hybrid library scenarios.

It may be noted that CERLIM has initiated some work in this area (MIEL4 to look at hybrid libraries - including, for example, the issue of comparing print and electronic equivalents - and MIEL5, designed to look at distributed services).

9.4 Agreement on international standards

There is already wide agreement (via ISO and IFLA) on performance indicators for traditional libraries. There is a need to encourage development of similar agreements on performance indicators for the electronic environment. CERLIM is leading a proposed study of this area (MIEL3) which has been submitted to the European Commission under the final Call for Proposals of its Libraries Programme.

9.5 Collecting data

A number of actions are needed under this heading. Firstly, work is needed to define methodologies to enable data to be collected economically and efficiently by each institution. Secondly, there is a case for considering how statistical data should in future be collected at national level. At present statistical data (e.g. on total book loans and total staffing in SCONUL libraries, leading to a national average) is collected by aggregating returns from all SCONUL members. For dataservices it would be more efficient to collect this data from the datacentres. At present the datacentres report this information to JISC, but in the context of their contractual service provision through formal annual reports rather than as a statistical collection exercise. There could be a role for a national agency, possibly by extension of the remit of a current agency, in this context or JISC may itself take on this role.

9.6 Understanding the operation of electronic libraries

It would be useful to explore site-based issues in some detail, preferably through detailed case studies. These would allow in-depth exploration of key concerns, including exploration of electronic library service quality (using the SERVQUAL and similar methodologies), identification of user behaviour in networked information usage, issues of service substitutability and, most importantly, impact studies (see below). 

9.7	The Impact of the Electronic Library

The consensus among the experts consulted during this Study was that there needs to be a new focus on the impact of electronic library services on their users. We know remarkably little about this subject. Staff and students in academic institutions make extensive use of electronic information resources, but we do not know which services are of the greatest value, nor how one service compares with another, nor, to any great extent, what it is about a service which gives it value (we posit that content quality is important, but know that interface design can be equally significant). For these reasons we recommend that a strand of work on the impact of the electronic library should be encouraged.

10. Contact Details

Further information on the MIEL Programme and on related work by CERLIM is available from:

			Professor Peter Brophy
			Head
			CERLIM
			University of Central Lancashire
			Preston
			PR1 2HE
			Tel: 01772-892261
			Fax: 01772-892937
			Email: p.brophy@uclan.ac.uk

C:\RESBIDS\MIEL\REPORT5.DOC

Appendix 1: Report of the MIEL1 Scoping Study

The Study Report is reproduced on the following pages.

UNIVERSITY OF CENTRAL LANCASHIRE

CENTRE FOR RESEARCH IN LIBRARY & INFORMATION MANAGEMENT

Management Information for the Electronic Library:

Report on a Scoping Study undertaken for the Joint Information Systems Committee under FIGIT’s Supporting Studies & Activities programme.

November 1995

CONTENTS

		Page	
1	Introduction	2	
2	Background	2	
3 	Current Work on Management Information and Performance Measurement	3	
4	Towards Management Information for the Electronic Library	4	
5	A Framework for Management Information Development	10	
6	Management Information for the Electronic Library: a possible model for development	12	

1	Introduction
	As part of the FIGIT Supporting Activities and Initiatives, a short ‘scoping’ study of management information requirements and provision for the electronic library has been undertaken. The intention of the scoping study was to identify the scope of a possible project which would provide the detailed framework for the development of management information systems and performance measurement in relation to the electronic library.

2.	Background
	All academic libraries in the U.K. have seen a considerable shift towards the use of electronic resources in recent years. Online services are, of course, well-established but until recently they formed a minor, albeit vital, part of the portfolio of services on offer. Over the last five years we have moved to a situation where online database services (both commercial and JISC funded externally-mounted services and locally mounted datasets, usually on CD-ROM) form a very significant part of the total service mix. The OPAC is now ubiquitous, and is being enhanced to provide additional services. Delivery of primary documents by electronic means is not unusual. The Electronic Libraries Programme (eLib) will itself give a boost to all these developments, especially as projects report and are implemented. On top of all this, electronic services are increasingly being delivered to the desk-top, outside the Library building and possibly off-campus. End-users may no longer require the services of either expert library staff, or access to a physical stock of materials, to make effective use of “library” services. In these circumstances:

	(a) Library staff may not know who is using which service, and may be ignorant of alternatives which users find for themselves

	(b) It is very difficult to know how much use is being made of the services provided by the library

	(c) The effectiveness of services is more difficult to judge.

	Indeed, some of our traditional management information and performance measures may be turned on their head. For example, the fact that a user comes into the Library may be a sign of failure (poor service) rather than success - she could not find what she wanted via the network, either because it had not been provided or because it was difficult to find and use. In such cases the Library itself was a poor second-best to the direct access the user had first sought. Similarly it has been noted that a drop in use of a dataset mounted at one site may indicate either unpopularity (fewer people using it) or greater popularity (caching at other sites to reduce traffic on the network).

3.	Current work on Management Information and Performance Measurement
	There is, of course, an enormous amount of activity related to the development of library management information and performance measurement. Within that corpus of work, however, there is surprisingly little on management information or on performance measurement for the electronic library. To give one example, in The Effective Academic Library  the discussion of the suggested measure ‘Documents delivered per student’ contains the comment: “as electronic delivery of documents becomes more common, it will be appropriate to add a number for this category to the basket (of document delivery types)”. The Report’s recommendations then note, “a measure for the delivery of electronic documents also needs to be established”. It is notable, however, that the recommended performance indicators remain heavily dependent on concepts of delivery based on print-on-paper - and while this may reflect current library practice, it will not be adequate for the future. The same comment could also be applied to the recently released draft of the ISO standard, the IFLA work on performance measures and the EC PROLIB study.

	Similarly, one of the most influential recent publications in the field of library performance measurement Measuring academic library performance: a practical approach  devoted very little attention to the issue of performance in the context of the electronic library. The measure termed ‘Total Materials Use’, for example, was defined as the sum of ‘Number of items issued, from statistics collected by circulation systems and any other sources’ and ‘Number of items used in the library but not issued, measured by counting the number of books needing to be reshelved’.

	Finally, by way of example, we might look at the COPOL/SCONUL statistics which have been collected for many years. Here statistics are collected on 'number of mediated online searches carried out' and 'expenditure on online searching'. Again, however, there is much greater emphasis on counting physical objects and events, rather than on activity related to electronic resources.

	There is a considerable amount of work being undertaken into software to support management information and decision making, notably through four parallel EC Libraries Programme funded projects: these are outside the scope of the present Study.

4.	Towards Management Information for the Electronic Library
	The above comments should not be taken to mean that there is no work of relevance to the development of management information systems for the library of the future. In this section, brief notes are provided on some of the work which could be drawn together as an input to the development of a systematic approach.

	4.1 Research studies The only detailed study which has been located is a US Department of Education project (Performance Measures for the Academic Networked Environment) being undertaken by Charles R. McClure and Cynthia Lopata at the School of Information Studies at Syracuse University. This work is ongoing, but an interim report was presented at the CNI Conference in Oregon in late October 1995. McClure and Lopata’s work is focused on the whole academic networked environment, rather than specifically on electronic library services, but is clearly of great relevance to the latter - and overlaps considerably with it. They suggest that performance measures should be developed in the following categories:

	Users: the number and types of users and the frequency with which they use the campus network;

	Costs: the total and types of financial resources that are expended to operate the academic network;

	Network Traffic: the amount and types of traffic flowing over the academic network, including router traffic, modem traffic and Internet traffic;

	Use: the amount and types of uses made of the network;

	Services: the applications and services that are made available over the network, including the OPAC, distance learning and user support. Also considered are: user satisfaction, the extent to which they are used, and the costs to provide them.

	In the discussions at CNI, McClure stated that it was realistic to expect measures to be developed which enabled longitudinal comparisons to be made within institutions, but that reliable inter-institutional comparative measures were a long way off.

	4.2 Automatic recording of activity Very many electronic services provide a log of activity, often subdivided by individual user or by user type. The reports issued to institutions by BIDS are a typical example, enabling librarians to identify use by department or individual. Such data can of course be presented in graphical format and used to monitor changes over time - for example it might be used to judge the efficacy of a training programme in a particular department. Similarly, most library systems log OPAC activity, and most CD-ROM networks can keep records of the number and length of uses of each dataset - although in his last case the capabilities of different systems vary widely: Silver Platter's ERL (Electronic Reference Library), for example, offers quite sophisticated data recording across multiple databases.

	4.3 Costs One of the ways in which electronic library activity is monitored throughout the world is by analysing the raw costs or the proportion of the budget which is being devoted to these services. Just as academic libraries often monitor the balance between expenditure on books and that on journals, so expenditure on different kinds of electronic service could be important for trend monitoring. Although crude, this can be an effective way of monitoring broad activity. Interestingly, it can also be interpreted as a reflection of the real priorities being pursued by managers.

	4.4 Commercial Suppliers An interesting approach would be to examine the developing practice of commercial suppliers of electronic information products, and especially their billing strategies. For example, UnCover charges per item requested, so needs a watertight mechanism for identifying users at that point - but has less interest in how the navigate to that point. Should libraries measure only a small, but highly significant, part of their services (a ‘critical activity’ approach), such as documents delivered, but not time spent online?

	4.5 Transaction log analysis There is considerable experience in the research community of using individual transaction logging to examine user behaviour and to draw conclusions concerning the type of use. These analyses vary from the overview (what proportion of searches use which index) to the individual (examining how individual users carry out their searches). The OKAPI work at City University has used this approach, as have some US researchers, and it was used in the EC BIBDEL experiment at the University of Central Lancashire

	4.6 Database quality The Centre for Information Quality Management (CIQM) is an agency set up by the UK Library Association and UK Online User Group (UKOLUG) to enable users to report any problems they encounter when using commercially produced databases. These comments are forwarded to the publisher, supplier or host as appropriate and the Centre then acts as a communications channel for any response. Another approach was represented by a recent study which looked at the quality of bibliographic records by examining use of the BLCMP database and checking how frequently records needed to be edited by member libraries before being added to their in-house catalogue databases.

	4.7 Practical Decision Making All librarians are being faced with making hard choices between traditional print-on-paper and electronic media, and often between different versions of the same publications. Some papers have been published on the criteria which can be applied and the information (on cost, usage, etc.) needed to inform decisions.

	4.8 Studies of specific services Pre-eminent among the studies of particular services, for our purposes, must be Harry East’s recent work on the use of BIDS-ISI. It will be important for librarians in the future to have a clear understanding of how services are being used and of their users' perceptions of the services on offer. More broadly, there would be merit in looking at the way non-library-mediated services are developing across the networks. For example, there have been several recent publications on charging systems which are predicated on the availability of high quality information on usage.

	4.9 International Standards Activity Although no international documents have been published recommending performance measures for the electronic library, it may be noted that the National Library of Canada made proposals to the ISO TC46/SC8 group for measures to be adopted. It is understood that the proposals were rejected by the group because they were felt to be premature and lacking international acceptance: a criterion for inclusion was that a measure should already have widespread acceptance, but this was clearly disadvantageous to development of measures of electronic services. It has been agreed that a further ISO Committee Draft (CD) will be distributed in December 1995 and this is expected to contain some performance indicators related to electronic services

	4.10 Reports from Electronic Dataservice Providers
	The annual reports from the providers of dataservices, especially those under the umbrella of JISC, provide useful insights into the management and other information deemed useful by providers. The JASPER Committee examined proposals recently and set up a working group to take the issue forward. Among the statistics suggested were:

	Online connect time (subdivided by institution, user, etc.)
	Maximum number of concurrent users
	Amount of data downloaded
	Hit rate
	System downtime
	Methods of access used (e.g. gopher, web)
	Helpdesk statistics

	While these are not exactly parallel with the electronic library situation, they indicate some of the type of statistics which may be needed. JASPER also noted that the statistics presented depend on the purpose for which they are needed, reflecting the library need for detailed information for operational decision-making and broader presentations for strategic decision making.

	4.11 User Satisfaction
	One of the key issues for any library is, and will continue to be, whether users are satisfied with the services on offer. In this respect there may be little change due to the nature of the service, although in-library surveys will be much less satisfactory. The electronic environment provides an opportunity to administer questionnaires electronically to all users, and outside the library sector this has been done quite frequently. The quality management research supported by the British Library and other work referred to earlier could provide useful input in this area.

	4.12 Electronic Library Research
	Among the wealth of projects being carried out around the world, there are some activities which could be drawn on to inform this study. Mention has already been made of McClure’s work in the USA and of the National Library of Canada's interest in this area: the National Library of Australia also has an interest (through the 'National Collection Access Strategy'). In the UK, the De Montfort ELINOR Project has been designed to provide management information on usage down to individual page level: this was ranked as one of the most important requirements when software was selected for the system. The British Library has established a joint working party with the Publishers Association on Statistics of Electronic Publishing. Among the US work on the field, the Digital Libraries '95 Conference (Austin, Texas, June 11th - 13th 1995) revealed some activity, although not focused specifically on this issue. For example, Nancy Van House gave a preliminary paper on the Electronic Environmental Library which included discussion of how system use could be evaluated. The Coalition for Networked Information (CNI) has identified management issues but not so far management information as a subject for working group activity. It appears that, to date, most work in the electronic library field relies on transaction logging of one type or another for its management information, but systematic attempts to provide a user (i.e. library manager) perspective on this have been lacking.

5.	A Framework for Management Information Development
	There are three main reasons for gathering and using management information in libraries, and these remain as valid in the context of electronic services as in the traditional library. They are: use for internal, operational management of services; use for forward planning, particularly of a strategic nature; and use for evaluation and review of services, including reporting to higher levels of authority. In each case trend analysis and modelling of decision making are important. It is useful to structure thinking about management information for the electronic library under these headings (while accepting that there is a degree of overlap).

	5.1 Internal Management of Services
	Clearly, much management information is needed for the internal management of the operational library. The equivalent of book issues (e.g. online accesses) may be used to rank the popularity of datasets so as to help decisions on purchase or removal. Information on equipment usage may inform decisions on replacement, upgrading and siting. Information on expenditure will be essential to financial management. In many ways this type of information, though different in kind, mirrors that currently collected for traditional materials - but with important differences which need to be explored.

	5.2 Forward Planning
	Information for forward planning will generally be at a broader level, but will again include usage and expenditure patterns, extrapolated to give a view on likely future trends. To this will be added information on the environment - ranging from intelligence on the sector as a whole and on library developments, to institutional plans, for example on student numbers. 

	5.3 Evaluation and Review
	All university libraries need to review and analyse their activities, and management information is vital to this process. The structure of The Effective Academic Library’s recommendations might form a useful basis for thinking through the management information needs of the electronic library in respect of evaluation and review. Within the five-fold structure suggested in that publication it would be possible to identify the 'electronic equivalents' or complementary measures which would apply to the electronic library. Sources of data could then be identified. Information to enable 'benchmarking' (such as the SCONUL dataset) would be vital to this process, although the earlier comments by McClure on the difficulty of devising measures which are truly comparative across institutions need to be kept in mind.
	
	It is suggested that these three uses of management information would each need to be informed by data generated locally (which would include the normal statistics gathered by university libraries as well as information on usage of locally-mounted datasets), by information from the use of external services where the library facilitates but does not 'provide' access, and by a national (and perhaps international) corpus of comparative data.	

	During the Scoping Study, a Workshop for senior academic librarians was held in London. The discussion at that event emphasised the need to embed management information for the electronic library into the context of institutional information strategies, and to develop approaches within the general framework of The Effective Academic Library. Data collection will involve specific studies (such as surveys, which may be automated as in systems like Libra), alongside data from library systems and service suppliers.

6.	Management Information for the Electronic Library: a possible model for development
	The expert Workshop was of the view that work on the development of management information and performance measurement for the electronic library was important and should be pursued. If FIGIT agrees that it would be valuable to continue the development of this area, it is suggested that a Project along the following lines should be commissioned:

	6.1 The conceptual framework described in Section 5 should be used as a basis for further work. This implies examining management information needs against each area, identifying sources of appropriate data and maintaining a close link with institutional information strategies.

	6.2 Each of the possible sources of information identified in Section 4 should be examined in detail and mapped onto the framework. There should be close liaison with the work at Syracuse University and at the National Library of Canada. Discussions should be held with key suppliers (both JISC services and commercial suppliers) and eLib project teams.

	6.3 A seminar for senior academic librarians should be held during 1996 at which interim findings should be presented and feedback sought: if possible contributors from North America should be invited to make presentations.

	6.4 A Report should be prepared suggesting key measures which could be adopted by institutions and detailing specific areas where further work will be needed.

	6.5 The work should take place in close liaison with the implementation of The Effective Academic Library.

November 1995

[ENDS]

Appendix 2:	Complete list of indicators for evaluation and review (EAL +)

This Appendix contains a list of the EAL indicators and the additional indicators suggested in Chapter 7 of this Report. Indicators suggested in Chapters 5 and 6 for operational management and forward planning respectively have not been included. The symbol  indicates new indicators for the electronic library: it should be noted however that nearly all other indicators will be modified to include aspects of electronic provision, as detailed in the text.

 P1 Integration
	 P1.1 Strategic Cohesiveness
	 P1.2 Resourcing Mechanisms
	 P1.3 Planning Processes
	 P1.4 Service-user Liaison
	 P1.5 Assessment and Audit mechanisms

 P2 User Satisfaction
	 P2.1 Overall User Satisfaction
	 P2.2 Document Delivery Services
	 P2.3 Information Services
	 P2.4 Study Facilities
	 P2.5 Information Skills programme
	 P2.6 IT infrastructure

 P3 Delivery
	 P3.1 Meeting service standards over a given period (local measure)
	 P3.2 Meeting development targets over a given period (local measure)
	 P3.3 Documents delivered per FTE student during a year (national measure)
	 P3.4 Enquiries answered per FTE student during a year (national measure)
	 P3.5 Proportion of students receiving post-induction instruction in information-handling skills during a year (national measure)
	 P3.6 Library study hours per FTE student during a year (national measure)
	 P3.6A   PC hours used per annum divided by FTE students
	 P3.7 Volumes in collection per FTE student at a given date (national measure)
	 P3-7A Proportion of JISC datasets available to users
	 P3-7B Total major electronic subscriptions

 P4 Efficiency
	 P4.1 Items processed per Library Staff FTE
	 P4.2 Total Library Expenditure per Items Processed
	 P4.3 Documents Delivered per Library Staff FTE
	 P4.4 Total Library Expenditure per Document Delivered
	 P4.5 Enquiries Answered per Library Staff FTE
	 P4.6 Total Library Expenditure per Enquiries Answered
	 P4.7 Total Library Expenditure per Study Hours p.a.
	 P4.7A Total library expenditure/PC hours used per annum
	 P4.8 Volumes in Stock per Library Staff FTE
	 P4.8A Total major subscriptions/FTE staff numbers
	 P4.9 Total Library Expenditure per volumes in stock
	 P4.9A Total expenditure/Total major subscriptions

 P5 Economy
	 P5.1 Total Library Expenditure per FTE
	 P5.2 Library Staff Expenditure and Operating Costs per FTE
	 P5.3 Space per FTE
	 P5.3A PC hours available per annum per FTE student
	 P5.4 FTE per number of libraries
	 P5.5 Acquisition Costs per FTE
	 P5.6 FTE per Professional Library Staff
	 P5.7 FTE per Seat
	 P5.7A FTE students per network PC

Appendix 3: MIEL Expert Workshop

As part of the Study an Expert Workshop was held in Preston on 16th May 1997. The Workshop Programme, Participants List and a tabled paper on ‘Key Issues’ are reproduced here.

I. 	Programme

10.00 - 10.30		Registration (Coffee available)

10.30			Introduction to the day, including the work of Professor Charles R. 				McClure (Peter Brophy)

10.50	SCONUL’s work and The Effective Academic Library (Jacqueline Whiteside)

11.10	Information Services in the Networked Environment: Recent Evaluation Efforts, Methods and Future Prospects.  A summary of recent developments in Charles McClure’s research (Peter Wynne for Charles McClure)

11.30	Coffee break

11.45			International perspectives: ISO (Alan MacDougall)

12.00			International perspectives:	 EC Initiatives (Shelagh Fisher)

12.15	The Management Information for the Electronic Library Programme [MIEL] (Peter Brophy)

1.00			Lunch

2.00			Round Table discussion: key themes (see separate paper)

3.15			Tea

3.30			Discussion continues

4.15			Summary of the day: future actions

4.30			Concluding remarks

II Participants

Name 							Institution

Christine Abbott				University of Birmingham
Jeremy Andrew					University of Central Lancashire
Ian Bloor					De Montfort University
John Blunden-Ellis				University of Salford
Peter Brophy					CERLIM
David Brown					St. Martin’s College
Geoff Butters					CERLIM
Zoë Clarke					CERLIM
Jenny Craven					CERLIM
Joan Day					University of Northumbria
John Dolan					Library and Information Commission
Harry East					City University
Shelagh Fisher					CERLIM
Geoffrey Ford					University of Bristol
Alun Hughes					UCISA
D. Larbey					University of East Anglia
Suzanne Livesey				CERLIM
Alan MacDougall				Dublin City University, Ireland
Rennie McElroy					Napier University
Anne Morris					University of Loughborough
Tony Oulton					Manchester Metropolitan University
Philip Payne					Leeds Metropolitan University
Don Revill					Liverpool John Moores University
Michael Roch					UCISA
David Spiller					University of Loughborough
John Sumsion					University of Loughborough
Jacqueline Whiteside				University of Lancaster
Peter Wynne					CERLIM

III Key Issues Paper

The Seminar has been structured with the afternoon session devoted to round-table discussions on key issues relating to management information and performance indicators for electronic libraries. It is to be expected that many of the issues raised in the morning presentations will inform these discussions. However, the following themes may also form part of the debate.

1	What are the issues which need to be resolved to provide reliable management information and indicators:

	 over time
	 between institutions
	 internationally

2	Is it possible to define a small set of key indicators which adequately represent overall performance?

3	What measures are needed to assess non-use - and the reasons for it?

4	Is it feasible to measure the performance of one service in a networked institution? Or should we accept that only institution-wide measures are feasible? Further, what are the institutional boundaries for such measurements?

5	How can the use of non-networked electronic resources (such as standalone CD-ROMs) be measured?

6	Is there a meaningful way of separating out information delivery from general use of communications systems like email and the web?

7	Is the concept of a “session” meaningful - or is there a better unit for measuring activity?

8	What additional measures and approaches are needed to deal with off-campus delivery and lifelong learning?

9	What steps can be taken to devise meaningful measures of user satisfaction when knowledge of services is uneven, skills often poorly developed and expectations unrealistic?

10	Should a national agency be set up to publish usage and other data on nationally-provided services and to undertake comparisons?

************************************************************

Further information on CERLIM’s research is available from

Centre for Research in Library & Information Management 
(CERLIM)
University of Central Lancashire
Preston
PR1 2HE
UK
Tel: 01772-892297
Fax: 01772-892937
Email: cerlim@uclan.ac.uk
Web: http://www.uclan.ac.uk/research/cerlim/