3. PROPOSED EVALUATION STRATEGY
A Formative and Summative Evaluation
A Composite Set of Recommendations
Main Evaluation Components
Phase One: Start-up
Phase Two: Formative Evaluation Activities
Phase Three: Summative Evaluation
This document contains the draft report of a brief consultancy (seven days) for FIGIT by the Tavistock Institute on the design of an evaluation of the Electronic Libraries Programme.
The brief of the consultancy, as agreed with the Programme Director, was to:
This draft report contains the proposals which will be presented for discussion at the FIGIT meeting on 11th and 12th July. A final report will be revised and completed in the light of the feedback received.
Initial Understandings
The Electronic Libraries Programme (eLib) is a strategic initiative in the domain of university libraries. It is intended to address developmental issues identified in the Follet Report, particularly the enhancement of services to researchers and teachers/students and reduction of costs through the application of information and communications technology.
The Programme (i.e. the Steering Committee, FIGIT) is already well advanced in terms of selecting and negotiating the content of projects, establishing a balance between areas, agreeing and implementing a consensus view on minimum criteria to fund projects, most notably on quality of management, credibility of consortia, additionality of funding, 'value for money' and 'sustainability' (beyond the funding lifecycle), as well as strategic relevance (beyond the immediate participating organisations), and disciplinary and institutional coverage.
The Programme is supported by a reasonably resourced, knowledgeable Directorate/Secretariat and a committee of key actors in the domain. It is worth noting however that the further ongoing contribution FIGIT committee members can be expected to provide may be difficult to sustain at present levels, in the medium term.
While the Programme has a functional workplan identifying a number of concrete areas of action-electronic journals, documentation services, training and network resources, projects have been selected to have implications and affects beyond their immediate objectives and institutional settings. All projects, to a varying degree, have agenda building, awareness raising, professional development (for librarians) aspects and many projects can be characterised as (formal or informal) action research interventions. More generally, projects can be characterised as practical experiments, and the programme as a whole as promoting systemic learning about new practices, opportunities and challenges within the professional and user 'community'.
While the programme is only one of a number of current initiatives - in Europe and in the world at large - it does take place in a clearly bounded system - UK Higher Education - which from, at least, the perspective of the Joint Funding Councils can be seen as a single institutional entity. Hence it is logical to appraise and value projects primarily in terms of system wide implications and affects. With a Follet preoccupation with 'courses and resources', i.e. meeting the needs of expanding university provision in a time of severe resource constraints, telematics is seen as a potentially cost effective way to expand services/facilities. The eLib programme is an instrument to accentuate and accelerate global processes and to reach and mobilise all parts of academia.
However, motivations for involvement differ for different programme stakeholders: for academics, the major programme interest lies in new ways of doing research and engaging in discourse; for universities, in providing expanded pedagogic resources for teachers and students with less resources; for libraries, cutting the spiralling costs of journal subscriptions and inter-library loans; for librarians, it is to address all these pressing needs (even crises), find new (expanded) roles, and to avoid redundancy of skills and functions; while for publishers, it is to protect existing markets and to develop new ones. Institutions and organisations participating in the programme do so to achieve particular, practical aims (an electronic journal, support for particular teaching needs), in addition to achieving associated strategic learning and positioning (about and vis-`-vis electronic journals, about and vis-`-vis electronic teaching resources etc.).
As in any process of technology development and application FIGIT is concerned to mobilise strategic actors, build on their existing capacities and help them develop new shared strategies, and collective agreement about long term strategies and new organisational arrangements. Interactions between user and research communities are complex and iterative particularly in HE where the communities are essentially one and the same. Network position and critical mass effect adoption and use as do standards and national and international policies. While FIGIT's establishment of sustainability as a criteria for project selection is laudable, inevitably the majority of projects will be overtaken by events and transmute in order to contribute to evolving agendas. The key catalytic affects of projects (in terms of changed orientations to action and network behaviour) may long precede the availability of functioning project products.
At this juncture, FIGIT is confronting the need to actualise the synergies and cumulative affects intended at project selection. FIGIT has to be somewhat reactive in terms of programme content. It is dependant ultimately on proposals which come forward, yet to ensure that the programme amounts to more than a 'basket of projects' requires that projects be aligned in practice with each other and with wider policy parameters and domain developments. This requires strong linkages between sub-sets of projects and the establishment of a common, mutually reinforcing, learning framework.
Ensuring such coherence at domain, programme and project levels is consistent with the strong collaborative traditions and partnership arrangements of HE libraries but will need to overcome some more autarchic tendencies in the HE sector more generally particularly as regards new institutional partnerships between libraries and teaching departments, HE and publishers, and sets of universities. Here again, ensuring systemic effects will depend on the degree to which projects can be successfully embedded and institutionalised within host organisations.
Implications for Evaluation
In summary, FIGIT's terms of reference and the contents of the Electronic Libraries Programme thus include:
In addition to project assessments which measure performance against original or revised objectives, each of these items suggests, and lends itself to, particular evaluation requirements. These are, respectively:
Beyond this, in devising an evaluation strategy for a decentralised innovative programme such as this it is normal to advise a significant concentration on evaluating programme architecture and implementation mechanisms (e.g. project selection, contract negotiation, monitoring and review, management, conflict resolution etc.) not least because it is difficult to interpret project and programme results without understanding the implementation processes that lay behind them.
However, this programme takes place in the context of well established conventional HE funding mechanisms, with well known strengthens and weaknesses. Any contribution evaluation can make, unless it aims to address a much wider arena than this individual programme, can only be marginal.
Assuming minimum levels of project control will exist along the usual lines of control/audit at the level of the contracting organisations, project steering committee's composed of individuals external to the project, and regular progress reviews (at least annual) by FIGIT,
A Formative and Summative Evaluation
The proposed evaluation strategy includes both formative and summative elements. As a formative evaluation, it aims to enhance and support the management of the Programme thus helping shape the Programme during its lifetime. However, it goes further than many formative evaluations by not only providing feedback to FIGIT management (Committee, Director, Secretariat) but also aims to support programme participants. It is assumed, for example, that the customers or stakeholders for the evaluation include the entire HE based UK library and information management community. More particularly, it is assumed that a formative evaluation will be a vehicle to mobilise domain actors, encourage cross domain learning and enhance the capacities within the community to manage change.
It has not proved possible to specify formative evaluation activities without also considering what such an evaluation approach cannot cover, i.e. what are the requirements for a summative evaluation. The proposed strategy therefore also includes the outline of a summative evaluation that will be able, by the end of the Programme, to provide evidence of results, achievements and outcomes. Such a summative evaluation will have to address policy implications as well as more direct project results and aggregate effects.
The entire evaluation strategy is intended to cover three levels of analysis: funded projects; the programme; and the domain - including key domain actors.
A Composite Set of Recommendations
Following the earlier discussion of choices to be faced in the evaluation design, this proposal takes the form of a composite set of recommendations. FIGIT/JISC may, of course, wish to select from among these recommendations and prioritise some rather than others. However, following discussions with the FIGIT Director, we are not now offering fundamentally different evaluation options for consideration.
Main Evaluation Components
The evaluation components fall into three phases, i.e. start-up, formative and summative. The 'task' description for each component is described below. A summary table outlining activities, expertise, resources and costs (where these can be estimated) follows these descriptions.
Most components are more or less discrete and can be individually commissioned and funded from a variety of sources.
Phase One: Start-up
Prepare Evaluation Framework
Given that the proposed evaluation strategy is made up of a number of components, there is a need for a common framework to ensure coherence. As a minimum, such a framework should prioritise the key evaluation questions, identify overall assessment criteria, lay out operational procedures and scheduling and specify links between evaluation activities. Such a framework need not be too complicated and should be reviewed on an annual basis, Much of the material for a framework already exists or is in the process of being elaborated. (For example, the 'Programme Success Factors' exercise on the 11th July added to the material gathered during the initial consultancy will provide most of what is needed on criteria).
Project Evaluation Guidelines
Generic evaluation guidelines have to be made available to projects as soon as possible. These should derive, in the first instance, from the evaluation framework already described. The guidelines should, however, assist with the structuring of both programme level and project specific evaluation considerations. This will require a process of consultation with all commissioned projects (a) to ensure that project evaluations are adequately accommodated and (b) to ensure that programme level expectations are realistic. There may also be resource implications for projects in fulfilling programme wide obligations in addition to project specific evaluative activities. Guidelines as envisaged will ensure that project level data can be analysed and integrated for programme purposes.
FIGIT Evaluation Infrastructure
A formative evaluation needs to be co-ordinated, utilised, integrated into programme management and serviced. To this end, we propose the following 'infrastructure' arrangements. (a) An evaluation subgroup of FIGIT should be designated; (b) an element of FIGIT secretariat time and resources, which may therefore need to be supplemented, should be allocated to evaluation implementation and support; (c) technical assistance regarding evaluation be provided to projects and other programme evaluation activities up to an agreed level. This proposal follows from the experience that without adequate support and management attention, formative evaluations will not be effectively utilised.
Phase Two: Formative Evaluation Activities
Project Level Evaluation
In line with their contracts and following the guidelines provided, projects will undertake their own evaluation activities. This will, at a minimum, involve the collection of data to answer key outcome questions, monitoring of implementation issues as they occur and synthesising results for project and programme purposes. For project purposes, clear links will be needed between project level evaluations and project management. For programme purposes, an annual report format, included within the proposed guidelines, will shape project reports on results, experiences and findings.
In each programme area (e.g. electronic journals) specific classes of information will need to be reported (e.g. for electronic journals on IPR strategy, technical standards adopted, researcher usage patterns, charging and licensing arrangements etc.). All projects will be obliged to assess their immediate and potential outcomes and benefits for participating organisations in appropriate terms-pedagogic, economic, socio-technical and so on. They will also be obliged to assess the strategic effects of the project on associated networks and periodically re-assess the project's likely sustainability and evolution.
Project level evaluations may variously be conducted by project staff, subcontracted experts and colleagues located elsewhere in the HE system.
Cross-Project and Cross Domain Evaluation Activities
Important outcomes of the Electronic Libraries Programme are likely to be located at domain level. Links will need to be established between key domain actors - many of whom will be involved in projects. Joint activities such as thematic evaluation studies across projects within areas may be required. (For example, many of the evaluation questions identified for investigation in the Super Journal project are equally relevant to the other electronic journal projects). These should be funded from a designated fund from which domain actors and appropriate outside experts will be invited to bid for small evaluation contracts. Involving project based actors and other members of the HE libraries and information management community will encourage direct involvement, diffuse experience and support membership by the community of relevant issues.
Networking, Communication and Concertation
In order to promote learning and mobilise the community, an active exchange and networking process is required. Although this is not an exclusive evaluation concern and such networking arrangements will be (are being) set up for various purposes, they are especially important in a formative evaluation. Thus, evaluation 'findings' - and problematics - need to be exchanged, methodological issues may need to be discussed and new consensuses will have to be developed. Occasional programme wide meetings and regular electronic communications - perhaps an electronic evaluation conference - are recommended.
Syntheses of Project and Inter Project Evaluation Outputs
In part due to the strategic nature of the projects selected, project learning will have relevance beyond the immediate project participants, With a consistent framework for project self evaluation a great deal of important learning will become available . If such knowledge is to be speedily accessible to the wider domain, rather than eventually seeping through the folk wisdom, it must be captured, systematised and disseminated.
Reflection at Programme Level of Evaluation Outputs
FIGIT itself will need to periodically collectively reflect on emerging evaluation findings and their implications, Annual review/reflection workshops should be scheduled in time with the cycle of annual project reporting and other evaluation activities.
Phase Three: Summative Evaluation
For reasons of accountability and credibility external rather than internal summative evaluation is appropriate, although such external summative evaluation will be structured by the core criteria established in the Evaluation Framework, informed by the formative evaluation elements outline above, and influenced (especially as regards policy) by FIGIT perspectives.
It should be noted that while summative evaluation normally takes place towards the end of life of a programme, some summative activities can take place much earlier. In this way summative components can also make a formative contribution to the programme.
Policy Mapping
Key objectives of FIGIT concern policy implementation and the link between IT developments and wider libraries' developments. We anticipate that programme outputs will include policy recommendations based on FIGIT experience and evaluation results. As part of the start-up activities a 'logical analysis' or mapping of programme activities in terms of programme objectives and related JISC and HEFCE objectives. Such a mapping will need to be periodically updated. It should provide base-line data on the extent to which programme activities are related to wider policy and institutional goals.
Assessing the Outcomes and Impacts of a Sample of Projects
The direct and indirect outcomes and impacts of a reasonably representative sample of projects will need to be assessed to demonstrate the range of programme benefits and to clarify the likely aggregate impact of the programme as a whole. Such an assessment should be based on, inter alia,
Establishing the Policy Outcomes and Implications of Programme Experience
The implications of the multiplicity of programme outcomes must be assessed and formally synthesised in terms of wider domain evolution. Policy recommendations can be formulated which translate programme learning into broader strategies and wider developments in JISC, and the HE sector as a whole.