7 Performance
evaluation
Introduction
- 7.1
- In embarking on the implementation of the public library networking
programme it will be necessary to build in processes to evaluate both
systems and services. Initially this will be in order to ensure that the
best value for money is obtained in both procurement and service
development. Subsequently the performance of what is being developed
will need to be monitored and evaluated so that provision can be adapted
and developed in ways which will have the maximum benefit. Ultimately
the users' views on what they get from the networked library will be the
most important in determining the future direction of the programme.
- 7.2
- In addition to the traditional methods of counting the use of
resources and services, the development of user surveys has considerably
advanced the understanding of perceptions and satisfaction levels among
library users. Many library authorities have supported recent national
initiatives led by CIPFA to establish standards in user surveys, for
both adults and children, and a UK picture of library user appreciation
is now beginning to emerge.
- 7.3
- Performance evaluation of networked electronic library services is
even more recent. Counting the number of accesses of a particular
electronic resource is straightforward, and 'Webwatch services' are
enhancing the capacity of the technology to monitor both usage and
users. However, evaluation beyond this level - testing the impact and
outcomes which result from use - is still at the development stage. In
this area, the public library networking programme will need to draw on
the research currently being undertaken in other sectors - especially
the higher education library sector. This chapter presents an overview
of the literature and the work currently in progress in this important
field.
Principles of performance evaluation
- 7.4
- In recent years interest in library performance measurement has been
intense, and various studies have been published on both sides of the
Atlantic. The reasons for this interest are not hard to find: pressure
on resources has led to an ever-more intensive search for efficiency of
operation, while concern to serve users' needs has focused attention on
effectiveness. Funders have demanded not only that value for money be
achieved, but that it be demonstrated by reference to factual data.
Users, and other stakeholders, have become more vociferous, while the
adoption of an 'access' strategy - using ICTs to reach remote resources
- in contrast to a 'holdings' strategy - the continual accumulation of
physical collections on site - has led to a greater reliance on external
providers, and, with it, greater use of contracts, service-level
agreements, etc.
- 7.5
- The performance indicators that are needed and how they are used can
be viewed from a number of perspectives. Policymakers, library managers
and customers will have varied attitudes to what constitutes an
efficient and effective public library service, although it is possible
to identify some commonly accepted indicators. Generally it is the
library manager who will need most the comprehensive approach, in order
to adapt services to meet needs and expectations. However, the
'stakeholder' approach does draw the attention of managers to the need
to demonstrate the value of the investment - it is axiomatic that the
user's perspective is, ultimately, the most important.
Performance indicators
for the networked library
- 7.6
- There has so far been no systematic study to develop and make use of
performance indicators for the new networked electronic libraries. There
are moves in this direction, however. The higher education community -
and the HE library community in particular - has amassed a wealth of
experience in developing networked services to the HE user community.
With this have come much debate on the role of the HE library in the
provision of electronic information and a recognition of the need for
new kinds of management information and performance indicators.
Electronic services are increasingly being delivered to the desktop, in
or beyond the campus.
- 7.7
- In making effective use of 'library' services, end-users will no
longer require access only to physical stock. In these circumstances the
contact between user and resource is invisible, and library staff may
not know who is using which service; they may be unaware of alternative
solutions which users find for themselves. The effectiveness of library
services is thus more difficult to judge than in a conventional
print-based environment. However, two recent research studies of
particular significance have focused on developing strategies for
generating management information and devising performance indicators
for (a) the academic networked environment and (b) the electronic
library.
The networked environment
- 7.8
- Assessing the Academic Networked Environment: Strategies and
Options (McClure and Lopata, 1996) is the outcome of a study
supported by the Coalition for Networked Information in the USA. The
publication is described as a manual which 'can assist network managers
and higher education decision-makers with improving the usefulness and
quality of their networks and ultimately increasing the satisfaction of
network users' (p. 3). Its authors:
- describe a range of techniques that assess the academic networked
environment;
- identify and discuss data-collection issues and problems;
- provide procedures for collecting and analysing the data needed to
produce the assessment;
- provide a baseline set of measures (e.g. counts of users, costs,
network services, support services, user satisfaction, etc.) for
conducting network assessments as a means for improving academic
networked services.
- 7.9
- Performance measurement allows the organisation to:
- Identify the successful and less successful aspects of the network
in relation to user needs and organisational goals.
- Provide trend data to assess changes in the use of the network and
network services over time.
- Assist decision makers in allocating (or re-allocating) resources and
in planning for future network developments.
- Assist network and library managers in justifying expenditure and
accounting for those expenditures.
- Monitor network activities and services to detect any change in
activities or the quality of services.
- Determine the degree to which users are satisfied with the network
and networked services.
- Serve as a first step in benchmarking, i.e. identifying best practice
performance, using that performance as a goal, investigating the factors
that led up to that performance, and then trying to replicate that level
of performance. (McClure and Lopata, 1996, p. 5)
- 7.10
- Assessing the Academic Networked Environment places a heavier
emphasis on qualitative methods of data collection than is usual in the
literature of performance measurement, and includes details of
techniques such as network benchmarking, focus groups, critical-incident
techniques, group process surveys, scenario development, and
observation.
The electronic library
- 7.11
- Management Information for the Electronic Library is a UK study in
progress for the Joint Information Systems Committee of the Higher
Education Funding Council, and is examining how performance indicators
for the electronic library might fit into the context of performance
measurement as discussed in the Joint Funding Council's Ad-Hoc Group on
Performance Indicators for Libraries report The Effective Academic
Library (HEFC(E), 1995). A range of indicators similar to that of
McClure and Lopata is being developed, but from a perspective of
information access and delivery. The study is being conducted at the
Centre for Research in Library and Information Management at the
University of Central Lancashire, following an earlier scoping study
(Brophy, 1995). The final report is due in summer 1997.
- 7.12
- Work in progress for this study (Brophy, 1997) demonstrates that
types of management information and performance indicators can be
identified in relation to library managers' needs for decision-making
information. Managerial tasks can be combined with identified functions
of the electronic library to provide guidance on the kinds of decision
which library managers will need to take to inform library planning and
ensure that outcomes represent the best value for investment and effort.
- 7.13
- The Management Information for the Electronic Library study is
highlighting the complexity of managing the ever-changing mix of
traditional and electronic services to deliver end-user services to ever
more diverse locations through the present-day academic library. The
need for defined and agreed performance indicators to evaluate these
services is an important issue.
- 7.14
- The networked public library will present the same challenges to
public library managers. Research in performance measurement and
performance evaluation in the networked public library will need to be
aligned with ongoing work in other sectors, and especially with that in
higher education.
References
Brophy, P. (1995). Management Information for the Electronic
Library: Report on a Scoping Study undertaken for the Joint Information
Systems Committee under FIGIT's Supporting Studies and Activities
Programme. Preston: CERLIM, University of Central Lancashire
(unpublished).
Brophy, P. (1997). Management Information Systems and Performance
Measurement for the Electronic Library: eLib Supporting Study. Draft
report to JISC/HEFC(E).
HEFC(E) (1995). The Effective Academic Library: A Framework for
Evaluating the Performance of UK Academic Libraries: A Consulative Report
to HEFC(E), SHEFC, HEFC(W) and DENI by the Joint Funding Council's Ad Hoc
Group on Performance Indicators for Libraries. Bristol: HEFC(E).
McClure, C. R., and Lopata, C. I. (1996). Assessing the Academic
Networked Environment: Strategies and Options. Washington, DC: CNI
Publications.
Report converted to HTML and hosted by
UKOLN
on behalf of the Library and Information
Commission.
Email technical queries on this website to
webmaster@ukoln.ac.uk
