DESIRE: Project Deliverable 	
Project Number:	RE 1004 (RE)		
Project Title:	DESIRE - Development of a European Service for Information on Research and Education		
Deliverable Number: 	D3.2 2		
Deliverable Title:	Specification for resource description methods Part 2: Selection Criteria for Quality Controlled Information Gateways		
Version Number	1.1 (Incorporates peer review comments)		
Deliverable Type:	PU		
Deliverable Kind:	RE		
Principal Authors:	Name	Paul Hofman, Emma Worsfold	
	Address	ILRT, University of Bristol, Bristol, BS8 1TN, UK.	
	E-Mail	paul.hofman@bristol.ac.uk	
	Telephone	+44 (0)1117 9289791	
	Fax	+44 (0)0117 9288473	
Other Authors:	Debra Hiom, Michael Day, Angela Oehler. 		
Deliverable URL(s):	http://www.ukoln.ac.uk/metadata/DESIRE/quality/report.rtf
http://www.ukoln.ac.uk/metadata/DESIRE/quality/		
Abstract:	
Selective subject gateways on the Internet are characterised by their quality control.  The core activities of resource selection and description rely on skilled human input (by librarians, academics and experts) and are not activities that lend themselves to automation.  

This report describes methods and tools that have been created to assist the staff of subject gateways to develop and maintain their quality control systems:

•	A generic conceptual model is provided, which describes the processes involved in running a gateway and the quality issues that relating to each part of the process.  The model can be used to monitor and review the quality of a gateway.  

•	A comprehensive list of selection criteria is given, which incorporates 'tips' and 'hints' for evaluating Internet resources.  This can be used as a reference tool for gateways interested in defining or refining the selection criteria their own use.

Both the model and the list resulted from  a 'state of the art' review of quality issues, both within subject gateways and in other sectors, notably the private sector and industry.  They have both been tested by a number of existing subject gateways.  The study and review carried out to develop these tools are described, together with the methods by which they were developed and tested.

Keywords:	information gateways, quality, subject gateways, selection criteria		

Contents
Contents	
PART II: Executive Summary	
PART III: Description	
Selection Criteria for Quality Controlled Information Gateways	
Introduction	
Aims and objectives	
Approach	
Desired outcomes	
The subject gateway model	
Background to the development of the model	
Defining a general model for subject gateways	
Results of the SSM analysis	
Pre-test model	
Field testing the model	
Test results	
The subject gateway model - conclusions	
Selection criteria	
Background to the development of the list of selection criteria	
Generating a comprehensive list of selection criteria	
Pre-test list	
Field-testing the selection criteria	
Post-test list	
The list of quality selection criteria: a reference tool for Internet subject gateways	
Conclusion	
A list of quality selection criteria: a reference tool for Internet subject gateways	
Contents	
1. Scope Policy	
2.  Content Criteria:  Evaluating the Information	
3.  Form Criteria:  Evaluating the Medium	
4.  Process Criteria:  Evaluating the System	
5.  Collection Management Policy	
Appendix I: Glossary - the subject gateways	
ADAM: (Art, Design, Architecture & Media gateway)	
BUBL: (The Bulletin Board for Libraries)	
EELS: (Engineering Electronic Library)	
EEVL: (Edinburgh Engineering Virtual Library)	
The Electronic Libraries (eLib) Programme	
NBW: (Nederlandse Basisclassificatie Web)	
OMNI: (Organising Medical Networked Information)	
ROADS: (Resource Organisation And Discovery in Subject-based services)	
RUDI: (Resource for Urban Design Information)	
SOSIG: (Social Science Information Gateway)	
Appendix II: Overview of Soft Systems Methodology	
1  Expressing the problem situation	
2  Rich pictures	
3  Root definitions	
4  CATWOE Analysis	
5  The Conceptual Model	
6  The five E’s	
7  Comparison	
8  Action	
Appendix III: Quality / selection definitions, models and methods in use	
Summary	
1. Introduction	
2. Concepts of quality in industry and management science	
2.1 Product quality	
2.2 Service quality	
2.3 Total Quality Management	
2.4 International standards	
2.5 Benchmarking	
3. Database quality	
4. The Internet and information quality	
5. Conclusions	
Appendix IV: Selection criteria of selective subject gateways	
Introduction	
1. ADAM	
2. EELS	
3. EEVL	
4. NBW	
5. OMNI	
6. RUDI	
7. SOSIG	
Appendix V: Selection criteria of other Internet services	
1 Introduction	
Argus Clearinghouse	
City.Net (excite)	
Cyberhound	
DINO	
Galaxy	
Jubii	
Lycos / A2Z	
Magellan’s Reviews	
NetFirst	
SUNET	
UK Web Library	
Webcrawler select (GNN select)	
WWW Virtual Library	
Appendix VI: Selection criteria found via the literature review	
On-line resources	
Print resources	
Selection criteria from the literature review	
Appendix VII: User surveys	
1. ADAM	
2. EELS	
3. EEVL	
4. OMNI	
5. RUDI	
6. SOSIG	
The 1996 survey of SOSIG users.	
Appendix VIII: E-Mail questionnaire sent to subject gateways, July 1996.	
Selection criteria issues raised by the subject gateways	
Appendix IX: Personnel involved in testing	
Appendix X: Rich pictures and models	
Part IV: Bibliography and references	
PART II: Executive Summary

This report stems from the preliminary work carried out on Indexing and Cataloguing within DESIRE. It is the second of three studies which look at current resource description approaches for selective subject gateways on the Internet. These three studies examine in turn: resource description formats (metadata), quality resource selection and finally classification. Each aims to independently provide a comprehensive and current snapshot of approaches in these areas.  As such they provide the basis for recommendations which will inform subsequent developments in DESIRE and be of general use to emerging selective subject services outside DESIRE. Each report has an associated set of recommendations, which itself does not form part of these reports, but exists as a working paper, updated throughout the life of the project.

The study detailed here has two foci. The first is narrow and specific and formed the initial motivation for this task - to define the quality criteria  to be used for resource selection within the DESIRE subject gateways. The second is far broader and stems from the need to provide a framework within which the quality processes necessary to a selective Internet service could be understood, developed,  tested and evaluated.

These two foci have produced two components of a toolset which can be readily adapted to suit the requirements of individual subject gateways and related services. To date, these two tools have been evaluated by existing subject gateways. They have been well received and appear to be flexible general purpose tools of value particularly to new and emerging services.

The first tool is a diagrammatic conceptual model developed using Soft Systems Methodology. SSM is particularly appropriate to examining problem situations (in this study the issue of quality for subject gateways) within purposeful human activity systems. The subject gateway model is not a specification or a system definition, but a conceptual representation of what is necessary to effect the principal transformations which subject gateways are established to achieve. The model is first and foremost an analytical tool to be used in a systematic comparison with the real world. The disparities which are observed in any such comparison are intended to result in real world action to improve the existing situation (continuous improvement).

The second tool is a comprehensive and structured list of quality criteria employed in resource selection. Subject gateways recognise that human judgement is essential if only resources of the highest quality are to be selected.  If detailed and definitive criteria could be established then expert systems could be developed to do the job, but this has not happened.  The implication is that the evaluation of information resources is a very complex process best carried out by subject specialists whose judgements are likely to involve detailed and complex mental processes.  It is necessary to draw out and formalise the tacit knowledge which is currently used so that the selection process becomes more transparent, consistent and accountable, and itself is subject to a process of continual improvement.

Both tools were developed independently of each other in the first stages. The model was constructed from an analysis of the intentions and expectations underlying the concept of a subject gateway, formulated as a series of root definitions. The list of quality categories and criteria was derived from an examination of what was happening in real selective Internet services. Only when the list had been analysed and grouped and the model had been drafted were the criteria mapped onto the model and the model itself revised to reflect natural clusters which had been found to exist in the real world quality criteria. 

It is important to recognise that as it stands this report is of limited general applicability to the wider community of potential providers of and participants in future selective Internet services. What is vital is that the core results (the tools) are developed during the lifetime of the DESIRE project and made available in a form which is of  maximum general use to that community. These tools are in essence the starting tools for the foundation of the cataloguing demonstrators in this workpackage. They will be further developed and refined  in parallel with the specification and construction of the Social Science demonstrator within this workpackage. Fully developed, they will form part of the Verified Toolset and Methodology (Deliverable D3.4).

PART III: Description
Selection Criteria for Quality Controlled Information Gateways
Introduction
This report is a study of the selection criteria employed for selective information gateways (e.g. subject gateways) on the Internet. It has been produced as part of the Indexing and Cataloguing activities for Workpackage 3, and concentrates on the subject gateways run by DESIRE (Workpackage 3) partners (EELS, KB, SOSIG) and the UK Electronic Libraries (eLib) Programme subject gateways (ADAM, EEVL, OMNI, RUDI, SOSIG) - see Appendix 1 for short descriptions of these services.  Selective gateways add value to Internet information because they can choose resources from the Internet with regard to subject matter or quality criteria. Libraries and librarians have an interest in this area and have been widely represented in the creation and maintenance of these services.
Aims and objectives
The aim of this task was to develop quality selection criteria and methods use by subject gateways.  This work may additionally be applicable to other selective Internet services, for example those primarily based on geographical or linguistic criteria.

The main objectives were to provide:

•	standards and procedures for resource selection by subject gateways.
•	methods (and tools) for the development of quality-controlled information catalogues.
•	quality mechanisms to permit the quality of information and information services to be monitored and reported.
•	a framework for quality assessment and control within subject gateways.
Approach
The study started with a ‘state of the art’ review, to capture materials and views in the areas of quality control and selection criteria.  The review covered:

•	quality models and methods being used in other fields, notably industry, management and information science
•	selection criteria and quality methods currently in use by subject gateways

The resulting studies can be found as appendices to the main report:

•	Definitions, models and methods of quality currently in use (Appendix III)
•	Selection criteria used by selective subject gateways (Appendix IV)
•	Selection criteria of other selective Internet services (Appendix V).
•	Selection criteria found by literature review (Appendix VI)
•	User surveys carried out by the selective subject gateways (Appendix VII)
•	E-mail survey of the selective subject gateways (Appendix VIII)

An initial review of literature and current practice indicated that the quality mechanisms in place for selective Internet services were fairly rudimentary when compared to developments in the commercial ‘customer oriented’ sector, where emphasis has been on developing systems of continuous improvement.  It was decided that a quality model designed specifically for Internet subject gateways should be developed, which would provide a framework within which to implement continuous improvement processes.  In parallel with the development of this conceptual model, detailed work would be done on the ‘resource selection’ process within subject gateways.  A comprehensive list of quality selection criteria was to be developed. This was to be mapped iteratively onto the subject gateway model.

The model and the list of quality criteria would be developed in the light of the ‘state of the art’, but would also be tested by existing subject gateways to ensure that they were of practical use in the field.  
Desired outcomes
The study aimed to generate two end products:

1.	A generalised graphical model of a functioning subject gateway that would enable a systematic approach to quality issues in the provision, development, control, monitoring and analysis of a subject gateway.

2.	A structured list of selection criteria that could be used as a reference tool by subject gateways and enable new and evolving subject gateways to produce their own tailored selection schemes without having to reinvent wheels.

The conceptual model aimed to be both comprehensive and generalised, not constrained to any particular subject area.  Derived from a ‘rich picture’ description of the activities necessary to achieve the objectives of a subject gateway, it would identify key points at which quality criteria are and may be employed; for example resource selection, training to enable users to make intelligent and informed searches, and quality requirements of information providers.

The list of selection criteria aimed to be comprehensive and flexible.  It would be a reference tool that subject gateways could use to assist the definition of the most appropriate criteria for their specific service.  As an internal test the list was to map directly onto the quality model, to enable services to apply quality mechanisms to the process of selection.

This report describes the background and evolution of these two products, and gives reference to the sources and testing used in this development.
The subject gateway model
Background to the development of the model
Subject gateways consciously emphasise the importance of skilled human involvement in the assessment and ‘quality control’ of their selected Internet resources.  The core activity - selecting and attributing meaning to those resources is a human activity.  Subject gateways are currently run as academic services and carry out activities that do not lend themselves to automation (recognising however the importance of complementary developments in automated resource harvesting to the growth of subject gateways).

From the outset  the view was taken that defining quality processes and criteria for a subject gateway involved more than simply listing the right questions to ask about potential resources. It was agreed that a  more rigorous framework was needed on which to hang the activities, processes and associated quality aspects.  Such a framework would  provide a useful tool for the specification, implementation, development and evaluation of  any subject gateway, making the substance of this report more generally applicable. It was hoped that in looking at quality from a broad base, such a model could prove valuable to a subject gateway as it could be used to:

•	stimulate discussion and aid problem solving by providing agreed points of focus
•	aid grouping and classification of various quality and control criteria 
•	identify possible quality improvement tools and methods
•	enable the testing of existing and subsequent specifications for subject gateways
•	help to identify and stratify training needs
•	provide a basis for specification and design to anyone setting up a subject gateway
•	provide for flexible development once established

Subject gateways use a technology (hard) system and a human activity (soft) system. The human actions need to be interpreted (understood) before designing or adapting technology that effectively supports the overall system. We decided to apply the analytical approach of Soft Systems Methodology (SSM) which would allow due consideration of both hard and soft aspects (Checkland 1981; Checkland and Scholes 1990).

See Appendix II for an overview of SSM and its key concepts.

SSM allows the development of a framework for collecting and interpreting information about the overall system, its associated issues and constraints. Ultimately it is used to define the overall system, its boundaries, the tasks performed by technology and by people, and how they interact. More structured, formal techniques can later be employed in specifying and designing the technical aspects of the service.

A model would be developed using SSM to build a framework for Internet subject gateways.  The methodology and evolution sections of this report describe how this was done.
Defining a general model for subject gateways
The University of Bristol’s Social Science Information Gateway (SOSIG) was used as the subject of the detailed study using SSM. SOSIG was a well-established real system (a subject gateway in action).  

The scope of the study was not restricted too early in the process by defining at the outset what the problem areas were.  The aim was to capture various individual views about purpose, goals or effectiveness of certain tasks or subsystems, the principal actors and clients involved, the transformations (what the system did) and what the expectations and constraints on the system were. Some of the approaches used were:

•	identify individual tasks performed
•	identify existing tools and methods
•	establish interactions between people and systems
•	make drawings of structures and layouts    
•	interviews: unstructured, informal (“tell me what you do”)
•	brainstorming issues / structured matrix
•	creative approaches to root definitions 

Information was mainly gathered from informal interviews, brainstorming and existing evaluation studies. This enabled the identification of relevant systems, current concerns and problems and concepts of what a subject gateway did and should be doing and the likely development issues. 

Rich pictures were generated to graphically represent the structure, processes and issues that could be relevant to the problem definition. In this case the rich picture rapidly developed into a generalised system model avoiding specific reference to SOSIG. (The distinction between the rich picture and the model distinction was not as easy to maintain as implied by Checkland).

Having gathered this information and structured a rich picture (graphics, text and an issues matrix) , a series of root definitions were written to express the primary and secondary objectives of the subject gateway. These definitions inform the iterative development of a graphical Conceptual Model of the overall system. A small  team worked on this so that missing information and conflicts in perceptions were highlighted. 

Subject gateways are usually established to fulfil a stated role and provide a certain service. We adopted a primary-task approach where an attempt was made to give a neutral account of the functioning of a subject gateway. We would looked at statements of what SOSIG was trying to achieve and portray to the outside world. The root definitions were written specifically as succinct statements that include Checkland’s CATWOE components.

It is important to note that the conceptual model produced is a theoretical construct - it does not represent the existing or potential structure of the organisation. A process of comparison or testing is required to link the conceptual model back to the real world.  It is this process of testing that will raise issues and imply subsequent action, both for the model (systems world) and for the subject gateway (real world).
 
Results of the SSM analysis
Rich Pictures
The rich pictures were produced as paper based drawings with attached notes and a matrix of current and anticipated issues  (see Appendix X for examples).  Attempts to make these more easily interpretable as rich pictures rapidly led to a production of the more idealised conceptual model.
CATWOE Analysis
Root definitions for a subject gateway were based on work within SOSIG. The CATWOE analysis revealed broad agreement over the individuals involved in the development and current functioning of SOSIG.  The following data were derived from informal interviews and discussions with directors, cataloguers, trainers, evaluators and users. The Social Science bias was subsequently removed for inclusion in the generalised conceptual model
Customers (who benefits)
Charities	FE students	
 HE students	Information support professionals	
Journalists	Librarians 	
Researchers in Government	Researchers in higher education	
Researchers in Industry	Researchers in NGOs	
Resource providers	SE students	
Self help organisations	Social science practitioners	
Statisticians	Subject specialist librarians	
‘Surfers’	Teaching staff	
Trainers	Undergraduates	
Actors
Advisory group	Cataloguers 	
Listeners	Other subject gateways	
Researchers	Software developers	
Subject specialist librarians	Systems administrators/managers	
Trainers	Trusted information providers	
User group		
Transformations

URL		URL with added value	
Vast		Small	
Unpredictable		Predictable	
Variable quality 		High quality	
Unmediated		Mediated	
Unstructured		Structured	
Users with poor search strategies		Users with well developed search strategies	
Unsafe environment		Safe environment	
Data		Data + meaning = information	
Timewasting 		Time efficient	
Labour intensive		Labour saving	
Information hungry		Enlightened /satisfied	
Internet unconfident		Internet confident	
No metadata record		Rudimentary metadata	
No subject sections 		Subject sections	
Reluctant		Enthusiastic (seeds interest)	
Researchers/users with habitual search patterns		Individuals with enlarged horizons of what is possible	
Intimidated users		At ease users	
No clues		‘now I know where to go’	
Uncritical information users		Users with well developed critical abilities	
Trainees		Trainers, Proselytisers, Sales force	
Information user and consumers 		Information  providers	
People requiring Internet presence		People with Internet presence	
Average career prospects		Enhanced career prospects for staff and users	
US biased		Reduced US bias	
Worldview
•	Information Superhighway - it’s the way to the future
•	Selectively mining information from an unmanageable data source confers economic and intellectual advantage.
•	The information available over the Internet needs to be controlled, moderated, systematised
•	Grey literature or incidental information
•	Ease of electronic publishing  allows the dissemination of different views - ones which publishers will not accept
•	Data is dynamic, immediate and pervasive - click on reload button after coffee break
Owners
These are the relevant bodies for SOSIG and simply appear as owners in the model

ESRC, eLib, JISC, DGXIII, University, Department, users
Environment
People, information overload, software constraints, funding, funding outlook.
Root Definitions
A university owned and maintained system that selects and catalogues subject specialist Internet resources on the bases of quality and relevance, allowing structured access by a range of users in research and education in the belief that such filtering provides an essential added value to the inadequately structured data available on the Internet.

A university owned and maintained system that introduces a range of users in research and education to the Internet as a potential source of relevant high quality information, allowing them to explore and develop discovery strategies which can be used in subsequent exploration in the belief that efficient and critical use of the Internet requires appropriate training.

An academic institution owned and maintained system that builds a publicly accessible catalogue of subject specialist Internet resources by the application of a predefined set of quality selection criteria.  
Conceptual modelling
These three root definitions provided different perspectives of emphasis for a particular subject gateway. They did not imply three different structures.  We attempted to move to a general model, independent of SOSIG, which would allow for all the above root definitions. 

The model went through three iterations before an agreed generalised model emerged, capable of incorporating all the above root definitions. These earlier iterations, the pre- and post- test models are detailed in Appendix X.

We looked at the logically necessary processes and components for each of the three root definitions at a depth which seemed to capture the system. We did not concentrate on establishing a hierarchical structuring of the processes, but preferred to work with the whole picture, recognising that for clarity and explanation we might want later to add such a decomposition.

A conceptual model was developed which represented graphically the activities logically necessary to achieve the transformation described in the primary root definition(s). The model was checked to ensure that it conforms to the following requirements:

•	represents exactly the activities (transformations expressed as verbs) required to achieve the goals of the organisation
•	meet the criteria for being a system 
•	is capable of being decomposed  hierarchically containing 5-7 activities at first-resolution level
•	have all components connected (except for monitoring and control units)
•	an ongoing purpose i.e. effects transformation
•	has provision for measure(s) of performance
•	has a decision-making or control process
•	consists of components (which are themselves systems)
•	exists as part of wider system, or environment with which it interacts
•	has bounded decision-making processes
•	has resources for its own use
•	an expectation of continuity
Pre-test model
See appendix II for a more detailed outline of  Soft Systems methodology and a definition of terms. The pre-test model is available at: <URL:http://sosig.ac.uk/desire/mdlv1_4.gif>
Field testing the model
An important stage in the evolution of both the list and the model was the testing, which had two main aims:

•	To collect test data that could be used to improve the model and the list
•	To get an evaluation of the practical value of the model and list in the field

Both products were subjected to testing and were modified in the light of the results.  

The testing was undertaken by the three organisations involved in building subject gateways as part of the DESIRE project; SOSIG, KB and EELS.  In addition to this, Biz/ed (an eLib funded subject gateway) took part in the testing due to their close working proximity to the SOSIG project.

A copy of the pictorial model and a set of the selection criteria were sent to each organisation.  They were asked to study the model and the associated textual descriptions and compare the processes indicated by the model against their own service.  Disparities between the systems model and the real world could indicate problems and/or where improvements could be made.

Methods for comparison that were used:

•	general discussion and observation: first impressions of disparities
•	question generation: the model was used to generate a series of  focused questions : Does the activity exist? Would it be useful? etc.
•	testing in practice: comparing what happens on a day to day basis in carrying out activities (resource description and cataloguing)
•	model overlay: comparing the conceptual model to the model implied by the organisation.

A similar procedure was used to test the quality criteria.  The participants were asked to work through the list criteria and mark down the criteria that they found relevant for use in their service.  See Appendix IX for a list of the personnel involved in the testing.
Test results
The final model takes into account some of the results of the testing that was carried out in the field. (NB some changes - notably those relating to heirarchical structuring of the model will be incorporated with the final version of this report). It is available as: <URL:http://sosig.ac.uk/desire/mdlv1_5.gif> . 
General impressions
1.	Well received
2.	The model was considered to be useful in general, particularly for the initial specification and design stages of a subject gateway, in the absence of any framework. (EELS, SOSIG)
3.	Mapped well retrospectively onto a new service (Biz/ed)
4.	It was seen to be useful in providing a framework for the decomposition of more complex problems (KB)
5.	It will be used in providing a framework for further developments of the NBW service (KB)
6.	Time consuming to understand
Process mapping
1.	High degree of mapping 
2.	Model indicated areas which would be of use
3.	Noted differences (restricted access), training users
4.	There were some noted changes and additions to processes
5.	Agreeing ‘select resource criteria’ (Activity B) to be used by distributed subject editors is difficult (EELS)
Feasible or desirable changes to the model
1.	Differentiate different data types (lists of criteria, on line help, resources) in a clear way on the model
2.	Provide a hierarchical overview - for clarity, display and reproduction
3.	Generate simpler model for smaller subject gateways (non-ROADS)
4.	It should be emphasised that the model is a conceptual, not organisational one
5.	There should be a more detailed examination of where quality criteria might be stated (i.e. criteria for the quality of quality of training materials)
6.	Add data record exchange between subject gateways
7.	Activity R (publicise) should feedback into the potential users' group (KB)
8.	Failed search or browse should result in suggestions to the user (KB)
9.	Failed browse should result in suggestions to use related categories in classification scheme (KB)
10.	Failed search - related keywords (KB)
11.	Ensure that user feedback results in further development of training material (KB)
12.	Users should be added to the group selecting potential resources (KB)
13.	Differentiate those activities which are automated and the which are human (KB)
14.	Use colours more to differentiate between parts of the model (KB)
15.	Provide a start and finish as it’s not immediately obvious how to read the model
16.	Ensure that it can accommodate the distributed nature of some activities
17.	Include and elaborate the process by which cataloguers, TIPs and Volunteers are trained
18.	More help and guidance through the model in the absence of personal one to one explanation

Most of these changes were subsequently incorporated graphically into the model of Appendix X.
The subject gateway model - conclusions
The generalised model that has been developed from this study aims to  provide the basis for a useful and comprehensive reference tool for Internet subject gateways. Used  in conjunction with the lists of quality criteria it will underpin the  specification and development of subject gateways within DESIRE. Additionally  the model attempts to provide a conceptual framework with which new, existing and emerging subject gateways and related selective gateways might evolve. It is not intended to be prescriptive and is not a system specification or design, but rather a means by which such specifications could be tested and developed.

Each subject gateway could make effective further use of the model if it were to:
•	define roles associated with each activity
•	define the 5 E’s criteria  (see Appendix II) for each activity
•	incorporate any additional necessary activities to fit into its principal aims
•	decompose of the activities into sub systems 
•	make explicit  the associated control criteria (scope policy, content criteria etc.)
•	formalise the review and monitoring processes associated with each activity

The model itself should evolve in use as the specification and development of a subject gateway proceed - it is not intended to be a static model. It is primarily a visual means to capture the complex functioning of a subject gateway in all its aspects. It has proved to be potentially useful in the evaluations carried out with a limited number of services. It should be used subsequently in the implementation phases of DESIRE to provide a means of structured evaluation of the service, establishing criteria for its effective and efficient performance. It will be produced in a revised format as one component tools and methods at the end of the project.
Selection criteria
Background to the development of the list of selection criteria
A initial review of the selection process currently being used by subject gateways revealed two key findings:

•	the selection is usually done by subject specialists (academics and librarians).  
•	many of the services have not formally developed or published any definitive selection criteria.

These findings reflect the premise on which these services are based - that human judgement is the critical factor if only resources of the highest quality are to be selected.  If detailed and definitive criteria could be established then expert systems could be developed to do the job, but this has not happened.  The implication is that the evaluation of information resources is a very complex process best carried out by subject specialists whose judgements are likely to involve detailed and complex mental processes.  It is necessary to draw out and formalise the tacit knowledge which is currently used in an unexamined way, particularly if the resource selection process is increasingly distributed as subject gateways enlarge and expand their information gathering activities.

The fact that selection is done intuitively and is based on human knowledge, experience and judgement raises the question as to what criteria this intuitive process involves.  This study aimed to gather as many of these criteria as possible, from a wide variety of sources, with a view to making as many of these criteria as possible explicit.  The list of criteria aimed to be comprehensive so that the benefit of this expertise could be shared as a tool for all subject gateways to use.
Generating a comprehensive list of selection criteria
A systematic review of selection criteria for Internet resources was conducted.  The initial aim was to capture all the selection criteria and quality attributes either currently being used by Internet services, or recently mentioned in the literature.  Four main information sources were used in the review:

•	Subject based selective services on the Internet	(See Appendix IV)
•	Other selective services on the Internet		(See Appendix V)
•	Related literature				(See Appendix VI)
•	User surveys from the subject based services	(See Appendix VII)

The aim was to produce a list of all criteria and attributes found.  Comprehensive coverage was the main aim.  The trawl was systematic, and at this stage all criteria were included in the terms that they were found in the sources, regardless of duplication or apparent value.
Pre-test list
Over 250 criteria were collected from the initial trawl.  This ‘raw data’ then went through the following processes:

•	Duplicates were removed
•	The language used was standardised.  The criteria and attributes were phrased as a question.  The question format was chosen because it was the most common format found during the review, and because it reflects the evaluative nature of the selection process.
•	A qualitative analysis took place.  The research team grouped the criteria thematically making use  of shared attributes and the quality model which was developed in parallel to this process.  
	
A second list was created by the research team and was to be tested by some subject gateways, after which any modifications necessary would be made

At this stage the list had been categorised into sections which aimed to reflect the different types of selection criteria, and the selection process itself.  These categories were designed in parallel with the quality model.  This list was then sent off to be tested by subject gateways (see Section 3.3).
Field-testing the selection criteria
An important stage in the evolution of both the list and the model was the testing, which had two main aims:

•	To collect test data that could be used to improve the model and the list
•	To get an evaluation of the practical value of the model and list in the field

Both products were subjected to testing and were modified in the light of the results.  

The testing was undertaken by the three organisations involved in building subject gateways as part of the DESIRE project; SOSIG, KB and EELS.  In addition to this, Biz/ed (an eLib funded subject gateway) took part in the testing due to their close working proximity to the SOSIG project.

A copy of the pictorial model and a set of the selection criteria were sent to each organisation.  They were asked to study the model and the associated textual descriptions and compare the processes indicated by the model against their own service.  Disparities between the systems model and the real world could indicate problems and/or where improvements could be made.

Methods for comparison that were used:

•	general discussion and observation: first impressions of disparities
•	question generation: the model was used to generate a series of  focused questions : Does the activity exist? Would it be useful? etc.
•	testing in practice: comparing what happens on a day to day basis in carrying out activities (resource description and cataloguing)
•	model overlay: comparing the conceptual model to the model implied by the organisation.

A similar procedure was used to test the quality criteria.  The participants were asked to work through the list criteria and mark down the criteria that they found relevant for use in their service.  See Appendix IX for a list of the personnel involved in the testing.
Post-test list
The final list was created in the light of the test results.

The list of quality selection criteria was well received as a tool for Internet subject gateways. All the testers gave a rating  4 or 5 on a five-point scale (where 5 was ‘very useful’ and 1 was ‘not at all useful’).  This result implied that drastic modifications were not required.  However, the ratings for individual selection criteria were used to make some minor modifications:

•	The number of items in the list was reviewed
•	The order of the list was changed
•	New criteria were added to the list

Only one of the criteria was not used by any of the services.  This was the Special Needs criteria in the Scope section.  None of the services said their users had any special needs that would affect the resources that were selected (e.g. disabled users requiring large print or audio resources).  In general the test results corroborated the idea that different services use different selection criteria, since there a variety of differing criteria were used by the different services.  

It was therefore decided that none of the items should be removed from the list, as they might be appropriate for some services.  However, the order of the criteria were altered, with the criteria used most commonly by the services doing the testing given a higher priority within each section (i.e. moved up the list-order).  

The testers’ comments in answer to the open questions gave a consistent picture of the relative importance of the different categories of criteria.  The scope criteria and content criteria tended to carry the most weight in the selection process of the majority of the services.  One service said the collection management criteria also carried most weight.  The process and form criteria tended to carry the least weight.

One new criterion was added to the list following a suggestion made by one of the testers.  It was suggested that ‘complementary value in a shrinking acquisitions budget’ should be used as a criteria.  This was added to the collection management criteria, as a valuable addition.  The complementary value of a resource in relation to traditional information resources available in libraries could conceivably affect the value of an Internet resource in the eyes of the users.

The test results were encouraging, in that they supported the idea that the list could be a useful reference tool for Internet subject gateways.  The comprehensiveness and adaptability of the list were well received. It was acknowledged that the list would need to be tailored to meet the needs of individual services to be of practical use:

‘I think it is useful to start out with this very comprehensive list, and choosing your priorities, work it down to something workable, and maybe from time to time reconsider your priorities by turning to the list once again.’ (A comment from the National Library of the Netherlands).
The list of quality selection criteria: a reference tool for Internet subject gateways
This list of quality selection criteria aims to be a useful reference tool for Internet subject gateways.  Its strength lies in the fact that it is:

•	comprehensive 
•	adaptable
•	organised according to the process of selection 

The list takes into account the fact that different quality criteria will be needed for different services, since ‘quality’ should be closely related to ‘user satisfaction’.  Different services will be aimed at different users, and so what constitutes a quality resource will vary across services.  The list aims to offer Internet services:

•	A generic framework in which to consider the quality selection process and quality selection criteria.
•	A comprehensive list of possible criteria which individual services can draw on to create or refine their own specific selection criteria
Selection criteria: a framework
As indicated in the quality model, selection is a process which involves careful consideration of a number of factors, all of which will affect the definition of a quality resource for the service.  The key factors in the selection process of an Internet subject gateway are generic: the users, the information resources, and the service itself.  The framework of the list takes all of these factors into account, by suggesting five main types of quality selection criteria:
1.	Scope Criteria:  			(Considering the Users)
2.	Content Criteria: 		 	(Evaluating the Content)
3.	Form Criteria:  			(Evaluating the Medium)
4.	Process Criteria:  			(Evaluating the System)
5.	Collection Management Criteria:  	(Considering the Service)

A ‘quality resource’ will therefore be defined with the specific service and its users in mind, as well as the nature of the information resources.  The quality selection criteria for a specific service can be created by using this framework.  Within each of the five areas the criteria most appropriate for the service should be decided, defined and continually reviewed.  The framework also helps to structure the actual process of selection:

•	Scope criteria will be defined at the inception of the service and will be the ‘first filter’ through which potential resources pass through. 
•	Content, form and process criteria need only be applied to resources that fall within the scope.  These criteria involve an evaluation of the resource itself.
•	Collection management criteria will take account of the coverage of the current collection, and may cause the other criteria to be changed or modified as the collection grows.

The framework accounts for the different stages at which decisions about quality need to be made.  Like traditional library collections, Internet collections involve selection, maintenance and de-selection.  This framework, in conjunction with the quality model, suggests that services need to apply quality selection criteria to resources at all three of these stages, and that many resources will need to be evaluated more than once, if the integrity of the collection is to be maintained.
Selection criteria: a comprehensive list to draw upon
All of the selection criteria found in the ‘state of the art’ trawl have been included in the list, and are organised according to the framework described above.  Individual services can use the list as a reference tool, to select the criteria that are appropriate for the service, in the knowledge that in doing so they will be drawing on a wealth of practice, experience and knowledge in this field.  

By using the framework, and drawing on the list, the definition of a ‘quality resource’ will be determined by the users and the aims of each service as well as by the nature if the resource.  The list can be tailored for use by any selective service.  The five main categories of criteria are generic, as they are based on the process of resource selection required to run any service.  Each service would need to select from the list, the criteria that are appropriate given their own particular user group and service aims.
Conclusion
The two tools which are detailed here will firstly be used in the specification and development of the Cataloguing demonstrators for the next phase of DESIRE. They will be further developed over the life of the project in parallel with the demonstration phase and be used to structure the subsequent evaluation. It is recognised that, having produced tools which are accepted as being of value to existing and emerging subject gateways, further effort will be required to make these generally useful and accessible  to new subject gateways which will emerge using the other tools and methods developed during the life of DESIRE.

A list of quality selection criteria: a reference tool for Internet subject gateways

For an interactive web version of this list see: 
http://sosig.ac.uk/desire/qindex.html
Contents

1  Scope Policy: Considering your Users

•	Information Coverage
•	Access
•	Cataloguing Policy
•	Geographical Issues

2  Content Criteria:  Evaluating the Information

•	Validity
•	Authority and Reputation of Source
•	Substantiveness
•	Accuracy
•	Comprehensiveness
•	Uniqueness
•	Composition and Organisation
•	Currency, and adequacy of Maintenance

3  Form Criteria:  Evaluating the Medium

•	Ease of Navigation
•	Provision of User Support
•	Use of Recognised Standards
•	Appropriate use of Technology
•	Aesthetics

4  Process Criteria:  Evaluating the System

•	Information Integrity (Work of the Information Provider)
•	Site Integrity (Work of the Web-Master/Site Manager)
•	System Integrity (Work of the Systems Administrator)

5  Collection Management Policy:  Considering your Service

•	Collection Coverage and Balance
•	Availability of Internet Resources
•	Availability of Library Resources

1. Scope Policy
The scope policy of a service states what is and is not to be included in the catalogue.  This is broadly defined at the inception of the service for the identified target audience.  

In the selection process, the scope of the service will affect the first decisions made about the quality of the resources.  Those falling outside the scope will be rejected, and those falling within it will go on through the rest of quality selection process, and be evaluated in the light of the rest of the quality criteria.

The scope criteria are the first filter through which the resources pass, and so are the most general criteria.  They will tend to involve black and white decisions - either a resource falls within the scope or it does not.  The most important thing to consider in choosing the scope criteria for a service will be the aims of the service and the target audience.

Information Coverage

Subject Matter

	•	What subject matter is appropriate for the target audience?
•	Are there any subjects which will be censored (e.g. for ethical reasons, such as resources produced by hate groups or resources about bomb-making/paedophilia)
•	How important is the subject matter of linked sites?	

Acceptable Types of Resource

	•	What types of resource are appropriate for the target audience? 
•	Is the information Scholarly rather than popular?
•	Does the resource contain more than just a list of links?
•	Is the site either proven to be, or expected to be durable?
•	Would a resource intended for use by an individual or local group be acceptable?
•	Is it innovative - does it make breakthrough design elements?	

Acceptable Sources

	•	Which sources of information are acceptable/appropriate for the target audience? 
•	Are academic, government, commercial, trade/industry, non-profit, private sources all acceptable?
•	Are pages maintained by individual enthusiasts (e.g. students) acceptable?	

Acceptable Levels of Difficulty
	•	What level of resource is appropriate for the target audience? (e.g. users may be school children or may be academics)
•	Is biased information acceptable, and are opinions and ideologies acceptable?	

Advertising

	•	Are resources that contain advertising acceptable?
•	Is there a limit to the amount of advertising that is acceptable?
•	Are there any forms of advertising which will be censored?	

Access

Cost

	•	How is charging going to affect selection - is the service only going to point to resources that are free to access?  
•	Are there any price limits in terms of the access charge?
•	What if resources are under copyright?	

Technology
	•	What technologies are appropriate for the target audience? (forms, ismaps, databases, cgi scripts, Java applications, frames, web sites, gopher, ftp, WAIS, telnet)
•	What connectivity does your audience have, and how will this affect selection?
•	What software do your users have and how will this affect selection? (E.g. will resources that work well in graphical-browsers but not in line-browsers be accepted?)
•	What hardware do your users have and how will this affect selection?	

Registration

	•	Will the service accept resources where user-registration is necessary before the resource can be accessed?  
•	Is on-line registration acceptable?
•	If users must negotiate written contracts before access is possible, is this acceptable?	

Security
	•	When it is necessary for users to send confidential information out over the Internet, will the provision of a secure coding system or encryption affect the selection?	

Special Needs
	•	Do your users have any special needs that will affect the resources selected?  Large print or audio options for disabled users.	

Cataloguing Policy

Granularity
	•	At what level will resources be selected/catalogued?
•	Will resources be considered at the web site/Usenet group level or the web page/Usenet article level?  	

Resource description

	•	What is the minimum amount of information needed to create a resource description in your catalogue?  I.e.  what basic information MUST a resource contain to be selected? (E.g. in a WWW document, contact details, last update details etc.)
•	Is there sufficient information to create a descriptive record (for your services ‘minimum set’)	

Metadata
	•	Will the service accept resources with/without specific metadata?	

Geographical Issues

Geographical Restraints
	•	Are any geographical restraints appropriate for your audience?
•	Will the service cover information produced locally, from particular countries, particular continents or world wide?	

Language
	•	Resources in which languages are acceptable/appropriate for your target audience?	

2.  Content Criteria:  Evaluating the Information
These criteria are based on the information content of resources.  The fact that the resources are Internet resources is not so relevant to the criteria in this section, indeed many of these criteria have been used by librarians in the selection of books and traditional information resources for many years.  

The criteria are listed in the main headings and the left hand column.  The right hand column contains some hints and checks that might be used to discern whether a resource meets with a particular criterion.

Validity

•	How valid is the content of the information?

•	Does the information appear to be well researched?
	
•	What data sources have been used? (The validity of these need to be evaluated -see sections below on authoritative and reputable sources)

•	Do the resources fulfil the stated purpose? 

•	Has the format been derived from another format e.g. print? (Is it an electronic version of a printed book/newspaper etc.).
	
•	Does the information claim to be unbiased (when in fact it’s biased?)

•	Are references given?
•	Is there a bibliography?

•	Does the resource have a scope statement?
•	Is there any information missing?
	
•	Is there any mention of the resource being available in another format?

•	Is the information what it appears to be?

•	Why is the information there?  What was the motivation of the information provider when they made the information available?  Do they have an ulterior motive?
	
•	Does the resource point to other sources which could be contacted for confirmation?

	

	

•	Is the content of the resource verifiable - can you cross check the information?

•	Is it merely vanity publishing?
•	Is there a request for payment?

•	The URL - Does the URL support the claim of authorship?
•	Email addresses given - Are emails for a publisher, the author, referees, sources, etc. given?  
•	Contact details given - Are addresses and phone numbers given that support claims of authorship, sponsorship etc.?
•	Traffic levels - Are they high?

•	Are the sources of the information stated?
•	Is it an often cited source?

Authority and Reputation of the Source

Who provided the information?

•	Is the source attributable to a reputable author or organisation?
	
•	Is the origin of the document documented?
•	Author’s title and institution/company displayed?
•	Is the information attributed to an author or editor?
•	Have you heard of the author before?
•	Is the URL a university server?
•	Is the author someone who has been cited frequently by respectable sources?
•	Do you know the educational background of the author?
•	Do you know the occupational background of the author?
•	Is the information written on a topic in the author’s area?
•	Is the author’s full name displayed?
•	Has the author had previous publications of note?
•	Is a biography of the author given?
	
•	Has the information been filtered?

•	How reputable are the ‘filters’?

	•	Is the site sponsored by a company, organisation or individual widely recognised as an authority or expert in the field?
•	Is the information peer-reviewed?
•	Has it been refereed?
•	Is the site sponsored?
•	Has the site been reviewed by a content reviewing agency?
•	What is the level of moderation for mailing lists and Usenet newsgroups
•	Via what source did you come across the resource? (i.e. did someone authoritative recommend it?)
•	Is there a common link to the page from a recognised authority?
•	Is the site linked to by multiple Internet sites?

•	Has the material been disseminated by a trade publisher?
•	Is the publisher known to you and reputable?
•	Is the publisher a recognised authority?
Is the publisher a ‘University Press’?

•	Is it by who it says it’s by? 
	
•	Can the authorship be validated?

	
•	Can claims to having been ‘filtered’ be validated?	
•	Can information located in the publication itself be used to determine the author’s credentials?
•	Is the author listed on say, the campus directory/organisational directory?
•	Are there email contacts for the publishers/referees/sponsors?	

Substantiveness

•	Is the information substantive?

	

•	Is there value added information?
	
•	Does the resource contain more than contact details?
•	Is the information full-text? (As opposed to just titles/bibliographic details)
•	Is it merely advertising?
	
•	If the resource consists of a collection of links is there substantial annotation or value-added information? (e.g. an annotated bibliography)	

Accuracy 

•	Is the information accurate?

	

•	Is the accuracy of the presentation adequate?	
•	Are you able to check the accuracy of the information?
•	Does the page cite a bibliography or provide references to confirm the accuracy of the information?
	
•	Is the grammar and spelling accurate?
•	Is there a prevalence of typographical errors?	

Comprehensiveness

What is the depth of the information?
•	To what level of detail does the resource go?
•	How superficial/exhaustive is the information?
•	Is some of the information incomplete?

What is the breadth of the information?
•	Are all aspects of the subject covered
•	Is everything you expect to find in the site there?
•	Are there any logical gaps of information?
	
•	Is the title informative?
•	Is an abstract given?
•	Is there an opening mission statement of the purpose of the resource?
•	Are there stated criteria for inclusion of information?
•	Does the index or contents page imply comprehensive coverage?
•	Are key words given that indicate the information content?

Uniqueness

•	Is the information on the site unique?

	•	Is it primary material?
•	Is there any original work available at the site?
•	Does the material have any relation to other works?
•	Is the site inward focused i.e. not just a list of links to external sites?
•	Is there any value added?	

Composition and Organisation

•	Is the information composed well?

•	Is the information clearly organised?

		
•	Does the text follow basic rules of grammar, spelling and literary composition?
•	Does it include jargon?
•	Is the information within a resource phrased unambiguously?
	
•	Is there a good structure?
•	Is the information within a resource arranged logically and consistently?
•	Is the information broken down into logical parts?
•	Is the resource well laid out?
•	Is the resource organised by the needs of the user?
	Is the information broken down into digestible parts?
•	Is the content clearly described?
•	Are the headings clear and descriptive?
•	Is there evidence of internal standardisation (e.g. use of a ‘style sheet’?)

Currency and Adequacy of Maintenance

(See ‘Resource Integrity’ section for details of this section)

3.  Form Criteria:  Evaluating the Medium
Form criteria are based on the presentation and organisation of the information.  Some of these criteria would be as applicable to paper resources as to Internet resources. Users need to be able to find their way around information objects,  however, the evaluation of Internet objects will involve checking electronic forms and so will involve different evaluative processes

Once again the criteria are listed in the main headings and the left hand column.  The right hand column contains some hints and checks that might be used to discern whether a resource meets with a particular criterion.

Ease of Navigation

•	Is it easy to navigate the resource?

•	Is it easy to browse the resource?

•	Is it easy to search the resource?

	
•	Are there hidden layers that are difficult to discover?
•	Does it take more than three ‘clicks’ (three links) to get to substantive information?
•	Do all the links serve an easily identified purpose?
•	Are all the links clearly labelled?
•	Do you ever find yourself in a position where there are no hyper-links to anywhere else?
•	Are hyper-links ambiguous i.e. is it obvious where a link is leading you to?
•	Do images support ease of navigation?
•	Are graphics/sounds/videos clearly labelled and identified?
•	Can pages or portions of a document be printed separately?
•	Are there single document options for those resources that may be printed?
•	For discussion in mailing lists and Usenet groups, are digests available?
	
•	Is there an index?
•	Is the resource indexed electronically?
•	Can a particular page be located from any other page?
•	Are there good back and forward links between pages?
•	Are the individual web pages concise or do you have to scroll forever?
	
•	Does the system have an effective search facility?
•	Is keyword searching possible?
•	How effectively can information be retrieved from the resource?
•	Is a well know search engine provided?
•	Does the search engine allow the use of Boolean operators?	

 Provision of User Support

•	Are there instructions?

•	Is there documentation?

•	Is there online Help?

•	Is customer support and training provided?

•	Do essential instructions appear before links and interactive portions?

•	Is there online documentation?
•	Is print documentation available?

•	Is there any online help?
•	Is contextual help available?

•	Is there an email ‘Help Desk’?
•	Is there a telephone helpline?
•	Are training materials/courses provided?

Use of  Recognised Standards

•	Are recognised standards used?
	
•	Is metadata provided?
•	Does it use standard multimedia formats? (e.g. MIME)
•	Is it written in standard HTML?
•	Have proprietary extensions to the HTML been added that some browsers will not recognise?

Appropriate use of Technology

•	How appropriate is the format?	
•	Does it do more than can be done with print?
•	Is appropriate interactivity available?

Aesthetics

•	Has consideration been given to the appearance of the site?
	
•	Does the resource follow good design principles?		
•	Does it look and feel friendly?

•	Is the balance of text, images, links, headers, font sizes and white space good?
•	Are the size, colour and animation of the images appropriate?

4.  Process Criteria:  Evaluating the System
Process criteria are based on the processes which exist to support the resource.  Unlike form and content criteria, these will be related closely to the fact that these are Internet resources.  The fact that information on the Internet lacks the integrity of a published work raises a number of questions about the quality of a resource over time.  The system that lies between information provision and information retrieval contains many variables, and some evaluation of these is needed to discern the quality of the resource.

The criteria are listed in the main headings and the left hand column.  The right hand column contains some hints and checks that might be used to discern whether a resource meets with a particular criterion.

Information Integrity  (work of the Information Provider)

•	Is the information current and up to date?

•	Is the information durable in nature?

•	Is there adequate maintenance of the information content?

	
•	If the site contains data or information that is time-sensitive, how current is this data and information?
•	How current is the material included in each update?
•	Is a date given stating when the web item was mounted?
•	Are time-sensitive resources available in near real-time?
•	Do the stated dates respond to the information in the resource?
•	Is the date given stating when the web item was created?
	
•	How time-sensitive is the information, and how does this relate to frequency of update? (e.g. for resources such as timetables, schedules and conference announcements)
•	If it is a static resource (not updated) will the information be of lasting use to the audience?
•	Is the information of a type that has a limited period of use?
	
•	Is the information provider likely to be able to maintain the information (unlikely in the case of information provided by students).
•	Is the resource improved and enlarged and updated appropriately?
•	Has the data been updated recently?
•	Is there a statement about the frequency of update?

Site Integrity (work of the Web-Master/Site Manager)

•	Is the site current and up to date?

•	Is the site either proven to be, or expected to be durable?

•	Is the site adequately administered and maintained?

	
•	Are there any dead links?
•	Are all the pages dated with the last revision date?
•	Are there links to sites that have moved?
•	Is a version number for the resource displayed?
•	Date of last update to the resource displayed?
	
•	Is there a description of the update frequencies for the resources?
•	Are you being redirected to a new URL?
	
•	Does the organisation or person hosting the resource seem to have the commitment to the ongoing maintenance and stability of the resource?
•	Is the site frequently updated/maintained?
•	Is the site regularly updated?
•	Are the downtimes announced?

System Integrity  (work of the Systems Administrator)

•	Is the technical performance of the resource acceptable?

•	Is the system stable?

•	Are adequate measures taken to maintain the integrity of the system?

•	Is the resource currently accessible?

•	Are the connections to the site providing the information reliable and stable?
•	Is it usually possible to reach the site or is it overloaded?
•	Are the downtimes infrequent?
•	Are the links reasonably stable?
•	Can you review the peak usage time for the resource (to assess reliability)?
	
•	Is the site mirrored?

5.  Collection Management Policy
The collection management policy of a service determines how resources will be selected or deselected in the light of the collection as a whole.  The term ‘collection’ refers to the items currently described in, and pointed to by, the catalogue.  

Collection Management involves de-selection (weeding) as well as selection.  The criteria listed below may be used to justify adding or removing a resource from the collection.  These criteria account for the fact that a resource may be selected at one point in time, whereas it would not be selected at another.  As the collection grows, the coverage and balance of the collection will change, and this may affect the selection process.

The criteria are listed in the main headings and the left hand column.  The right hand column contains some hints and checks that might help to discern whether a resource meets with a particular criterion.

Collection Coverage and Balance

•	What’s already in the collection?
	
•	What is the relative value of the resource in comparison with others already in the collection?
•	Is the information unique within the context of the total collection or does it duplicate?

•	Do a search to avoid duplication
•	Browse to see which areas are well covered/where gaps in the collection are	

Availability of Internet Resources

•	What’s available outside the collection?

•	What is the relative value or the resource in comparison with others available on this topic?  Is there similar/better subject material available?
•	Does the site contain information that is not readily available: would you have to look long and hard to find similar data?

•	Look for other resources with value added to the information	

Availability of Library Resources

•	What’s available via other information sources?

•	Does the resource provide access to information that is not easily obtained in other formats/not held in many libraries?
	Does the resource provide access to information that users may not be able to find/get access to in libraries?		

•	Look for resources which compliment existing information/library collections.
•	Look for electronic versions of texts that are in great demand in libraries in paper format, and therefore, difficult to get hold of.	

Appendix I: Glossary - the subject gateways
ADAM: (Art, Design, Architecture & Media gateway)
ADAM (Art, Design, Architecture & Media gateway) has been funded under the eLib programme to provide an “information gateway to quality-assured resources” in its subject area (ADAM 1996).  It is a 3 year project, and commenced in August 1995.

<URL: http://adam.ac.uk/>
BUBL: (The Bulletin Board for Libraries)
BUBL (The Bulletin Board for Libraries) was created in 1990 when it was part of the Glasgow based Project Jupiter.  Following a period of time when it was run on a voluntary basis, since 1994 it has received funding from the Joint Information Systems Committee (JISC) of the UK higher education funding councils.  BUBL’s original service was intended to serve as a bulletin board for the academic library community (Nicholson 1993).  Indeed, in a review of BUBL in Ariadne, Traugott Koch described it as “the Number One Internet resource for librarians” (Koch 1996).  However, as BUBL was being used by the wider academic community, its scope was expanded to cover other subject areas and it is now explicitly funded by JISC to provide a subject-based information service.  BUBL’s original subject-tree approach has now been replaced by a new subject-based approach called LINK (Libraries of Networked Knowledge).  LINK classifies all resources in the Dewey Decimal System.

<URL: http://catriona.lib.strath.ac.uk:80/ISC0>
EELS: (Engineering Electronic Library)
EELS (Engineering Electronic Library, Sweden) is a co-operative project of the Swedish University of Technology Libraries and its purpose is to provide an information system for quality assessed engineering resources on the Internet.

<URL: http://www.ub2.lu.se/eel/>
EEVL: (Edinburgh Engineering Virtual Library)
EEVL (Edinburgh Engineering Virtual Library), an eLib Project based in the UK, is an attempt to “build a gateway for the higher education and research community to facilitate access to high quality information resources in Engineering” (EEVL 1995).  It is a 3 year project, and commenced in August 1995.

<URL: http://eevl.icbl.hw.ac.uk/>
The Electronic Libraries (eLib) Programme
Some of the subject gateways (ADAM, EEVL, OMNI, RUDI and SOSIG) have been funded under the UK Electronic Libraries (eLib) Programme.  The eLib Programme was funded by the UK higher education funding councils in response to the report of the Joint Funding Councils’ Libraries Review Group (1993), known as the “Follett Report”.  The eLib Programme is managed on behalf of the funding councils by the Joint Information Services Committee (JISC).  The subject gateways funded by the eLib Programme are part of an Access to Network Resources (ANR) section, designed to “encourage the development of networking navigation tools and the growth of local subject based tools and information servers” (JISC 1994).  

<URL: http://www.ukoln.ac.uk/elib.html>
NBW: (Nederlandse Basisclassificatie Web)
The Nederlandse Basisclassificatie Web (NBW) or Dutch Classification Web is a retrieval system for Internet resources classified relevant to the Dutch academic community.  It was set up by the Koninklijke Bibliotheek (The National Library of the Netherlands) and has been further developed by co-operation with other Dutch academic libraries.

<URL: http://www.konbib.nl/basisclas/basisclas.html>
OMNI: (Organising Medical Networked Information)
OMNI (Organising Medical Networked Information), an eLib Project based in the UK, is specifically concerned with ‘high quality information’ concerning all aspects of medicine.  OMNI is a 2 year project which began in the summer of 1995.

<URL: http://omni.ac.uk/>
ROADS: (Resource Organisation And Discovery in Subject-based services)
ROADS (Resource Organisation and Discovery in Subject-based services) has been funded by the eLib programme to design and implement a user-orientated resource discovery system.  The ROADS software (Knight and Hamilton 1996) is used by several of the eLib Programme subject gateways.

<URL: http://www.ukoln.ac.uk/roads/intro.html>
RUDI: (Resource for Urban Design Information)
RUDI (Resource for Urban Design Information), an eLib Project based in the UK, is investigating the issues surrounding “the creation and development of significant multimedia information resources for research and teaching mounted on wide area computer networks” using the subject area of urban design.  The project intends to build up a collection of hypermedia materials on urban design - mostly stored on its own server.  RUDI began on the 1 January, and is scheduled to run for three years.

<URL: http://rudi.herts.ac.uk/>
SOSIG: (Social Science Information Gateway)
SOSIG (the Social Science Information Gateway) attempts to help the process of Internet navigation by collecting and organising social science resources.  It was begun with funding from the Economic and Social Research Council (ESRC) to provide an information gateway to social science resources on the Internet, and has also been funded as an eLib project.  It has recently received additional funding from both the ESRC and JISC to extend the project until 1998.

<URL:http://sosig.ac.uk/>

Appendix II: Overview of Soft Systems Methodology

The Soft Systems Methodology (Checkland 1981; Checkland and Scholes 1990) enables a structured approach to a situation that is seen to be problematic or where improvement is desirable. It consists of seven principal stages,  not necessarily employed sequentially and employed for systems and identified subsystems.

1	Problem situation unstructured		
2	Problem situation expressed	Rich picture	
3	Formulating Root definitions	CATWOE root definitions	
4	Conceptual modelling	Conceptual systems model	
5	Comparison system/real world	List of discrepancies	
6	Feasible/desirable changes	List of changes	
7	Action to improve	List of actions                    	

1  Expressing the problem situation
The scope of the study is  not restricted too early by defining at the beginning what the problems are. The aim is to capture primarily who individuals feel are the principal actors and clients of the organisation, what transformations are carried out and what expectations and constraints on the system were. This can involve:

•	identifying the tasks performed
•	identifying  tools and methods employed
•	establishing  interactions between people/systems
•	making drawings of structures/layouts    
•	conducting unstructured and informal interviews: (“tell me what you do”)
•	brainstorming issues
•	encouraging a creative approach to formulating root definitions (what does the organisation do?)
2  Rich pictures
Rich pictures are generated to represent structures, processes and issues of the organisation which could be relevant to the problem definition, and to try to give an impression of the organisational climate. Rich pictures are usually, but not exclusively, a mixture of  text and graphics and are intended to give an easily intelligible image of the organisation and the major issues involved. The production of a rich picture is not the purpose of SSM but can be viewed as a by-product of the process of investigation (Lewis 1992).
3  Root definitions
Root definitions are concise textual definitions that express the nature of the systems relevant to exploring the problem situation. They take the form: do X by Y to achieve Z. Having collected information to work with, a series of ‘root definitions’ are written to express the primary and secondary objectives of the system. These textual definitions inform the iterative development of  a subsequent  ‘conceptual model’ of the overall system.  It is important that this formulation occurs as a team activity, so that  missing information and conflicts in perceptions are highlighted

It is important that more than one root definition is formulated as each one expresses a different perspective of the organisation’s purpose and serves to acknowledge that there are conflicts and problems between (for instance) actors, owners and clients of a system. Primary task definitions aim to be a neutral account of what is necessary for the organisation to fulfil its primary role, sometimes from a mission statement, charter or official description of the organisation’s activity. Another approach is to seek agreement amongst the problem owners as to what the primary-task root definition is. Issue based definitions frequently look at part of the system only and focus on key problem areas.
4  CATWOE Analysis
The root definitions are written specifically to include Checkland’s CATWOE components. These are: Clients, Actors, Transformations, Worldview, Owner, Environment (TWECOA is considered to be a more appropriate ordering).  Describing the transformation and worldview are essential for the construction of the conceptual model. 

•	Transformations: all the changes which the system brings about, stated as an input andoutput.
•	Worldview: how an organisation is perceived by the clients, actors and the outside world. Different groups will have different world views. It is important to highlight any conflict between worldviews as these are likely to manifest themselves as current or future problems for the organisation in question.
•	Environment: things that influence but do not control the system under study. 
•	Clients: those who benefit from or are affected by outputs (products, services) of the system. 
•	Owners: individuals and organisations who could cause it to cease to exist
•	Actors: those who carry out activities within the system
5  The Conceptual Model
A (graphical) ‘conceptual model’ is then developed which represents graphically the activities logically necessary to achieve the transformation described in the primary root definition. The model is checked to ensure that it conforms to the following requirements:

•	it represents exactly the activities (as verbs) required to achieve the goals of the organisation
•	it meets the criteria for being a system
•	it decomposes activities in a hierarchy containing  5-7 activities at first-resolution level
•	it has all components connected (except for monitoring and control units)
•	it has an ongoing purpose, i.e. effects transformation
•	it has provision for measure(s) of performance
•	it has a decision-making or control process
•	it consists of components (which are themselves systems)
•	it exists as part of wider system, or environment with which it interacts
•	it has bounded decision-making processes
•	it has resources for its own use
•	it has an expectation of continuity
6  The five E’s
These are the criteria by which the transformations within the system are judged:

•	Efficacy - does the means employed work? 
•	Efficiency - is resource use minimised?
•	Effectiveness - does the transformation help to attain the long term goals?
•	Ethicality - is the transformation a moral thing to do?
•	Elegance - is the transformation performed aesthetically?
7  Comparison

It is important to note that the conceptual model produced is a theoretical construct - it does not represent the existing or potential structure of the organisation. A process of comparison or testing is required to link the conceptual model back to the real world. It is this process of testing which will raise issues and imply subsequent action.

Disparities between  the systems model and the real world might indicate problems and/or where improvements could be made. The output from this stage of testing is a list of system activities, the corresponding activities in the real world, and the differences.

Methods for comparison:

•	General discussion and observation: first impressions of disparities
•	Question generation: the model could be used to generate a series of  focused questions: does the activity exist? Would it be useful? etc. Issues of:  how can its performance be measured? Effectiveness? Efficiency?
•	Testing in practice: comparing what happens on a day to day basis in carrying out activities.
•	Model overlay: comparison between the conceptual model and the model implied by the organisation.
8  Action

Issues raised in testing should be addressed and actions which bring improvements should be identified and implemented. This could  necessitate changes to structure, procedures and attitudes. Proposals should then be drawn up stating what should be done and how :

•	decide what needs to be done, stating clearly the aim or objective
•	determine alternative ways of achieving the objective
•	appraise the costs of each alternative
•	build a model (if required) of the different alternatives, and test each model under different conditions
•	decide, on the basis of pre-defined criteria, the preferred or optimal alternative.

Appendix III: Quality / selection definitions, models and methods in use
Summary
Quality can be applied to many products and processes.  Traditionally it has been applied by management scientists to refer to product quality and, more recently, service quality.  Although it can be difficult to adequately define “quality”, current approaches to quality stress that processes are as important as tangible results, as is the case with TQM.  Standards like ISO 9000 and BS5750 provide a framework for the implementation of quality management in an organisation, but do not otherwise address the issues of product or service quality.  The information industry has recently turned its attention to the notion of product quality with regard to online databases and CD-ROM and progress has been made in co-operation with the library community and online user groups.  Quality is usually defined in relation to a set of guidelines or criteria.  The same broad approach is currently being applied to information provided over the Internet.
1. Introduction
Internet subject gateways have mostly defined “quality” with relation to carefully chosen lists of selection criteria. Quality has, however, been a subject of serious study in industry and management science since the Second World War. It might be useful to describe some of the concepts developed by management theorists and practitioners and to investigate their relevance to the quality selection issue.  Finally, work being done on databases and information quality will be described.
2. Concepts of quality in industry and management science
Quality has been analysed as a factor in the management process since the 1930s, but it was not until after the Second World War that it became important.  North American managers brought in to advise Japanese companies on restructuring after the war devised new concepts of quality which began to be accepted as being of universal application.  The important pioneers in this field were W. Edwards Deming, Joseph M. Juran and Kaoru Ishikawa.

A universally agreed definition of quality still does not exist. Juran (1988, p. 2.8) suggested that quality should be seen as “fitness for use”. Another short definition views quality as “conformance to requirements” rather than “goodness, or luxury, or shininess, or weight” (Crosby 1979, p. 17). These definitions from the management literature make it clear that quality cannot just be defined in relation to some abstract concept of “excellence”, but should be seen in relation to the demands of the user of the final product. A recent working definition of “quality” has been provided by Clark, Money and Tynan (1990, cited in Clark 1992):

“How consistently the product or service delivered, meets or exceeds the customers’ (external and internal) expectations and needs”.

In the management context, quality processes can be applied to any product. A product can be defined as the “output of any process”, consisting mainly of “goods, software, and services” (Juran 1988, p. 2.2). 
2.1 Product quality
Product quality is usually defined with specific relation to the product, whether it is a good or service. For goods, important aspects might be reliability, durability, performance characteristics, aesthetics, etc. These dimensions will differ according to the product type: the most important factor being whether it meets the end requirements of the customer (Bergman and Klefsjö 1994, p. 19).
2.2 Service quality
Early quality models concentrated on goods. The enormous growth of the service sector in Western economies since the Second World War has resulted in a growing literature on service quality. Defining and modelling the quality of services is generally acknowledged to be more difficult than modelling the quality of goods due to  the intangible nature of services themselves (Bergman & Klefsjö, 1994, pp. 266-267). There are two popular models of service quality in use.

1. Grönroos’s service quality model
The model created by Grönroos (1984b) attempts to understand how the quality of a given service is perceived by customers. It divides the customer’s perception of any particular service into two dimensions:

1.	Technical quality - What the consumer receives, the technical outcome of the process. 
	
2.	Functional quality - How the consumer receives the technical outcome, what Grönroos calls the “expressive performance of a service” (Grönroos 1984b, p. 39). 

Grönroos (1984b, p. 41) suggested that, in the context of services, functional quality is generally perceived to be more important than technical quality, assuming that the service is provided at a technically satisfactory level. He also points out that the functional quality dimension can be perceived in a very subjective manner  (Fig. 1).

Fig. 1.	Grönroos’s Service Quality Model

					
Expected	Perceived Service Quality	Perceived	
Service				Service	
					
		Image			

				
	Technical
Quality		Functional
Quality		
					
	What?		How?		

Source: Grönroos (1984, p. 40)

Grönroos’s model is important because it reminds us that service quality must include the manner in which it is delivered.

2. The ‘Gap’ model
The ‘Gap’ model (Parasuraman et al. 1985; Zeithaml et al. 1990) is a means of describing customer dissatisfaction in the context of service quality. A team from Texas A&M University carried out some interviews with executives in U.S. firms and with consumers. A series of five ‘gaps’ regarding service quality were then identified:

“A set of key discrepancies or gaps exists regarding executive perceptions of service quality and the tasks associated with service delivery to consumers. These gaps can be major hurdles in attempting to deliver a service which consumers would perceive as being of high quality” (Parasuraman et al. 1985, p. 44).

Five gaps were identified: 
1.	Between customers’ expectation and management’s perceptions of those expectations, i.e. not knowing what customers expect 
2.	Between management’s perceptions of customers’ expectations and service quality specifications, i.e. the wrong service-quality standards. 
3.	Between service quality specifications and service delivery, i.e. the service performance gap. 
4.	Between service delivery and external communications to customers about service delivery, i.e. when promises do not match delivery. 
5.	Between customers’ expectation and perceived service (the total of the other four gaps). 

It is this last ‘gap’ which has the most significance. The ‘Gap’ model keeps a clear focus on the perceptions of the customer, and these are seen as paramount

As part of this research, criteria for evaluating service quality were gathered. Ten key categories were identified which they called “Service Quality Determinants”, and noted that despite the different types of service analysed, consumers used fairly similar criteria. The ten Service Quality Determinants listed by Zeithaml et al. (1990, pp. 21-22) were the following: 

•	Tangibles - Appearance of physical facilities, equipment, personnel, and communication materials. 
•	Reliability - Ability to perform the promised service dependably and accurately. 
•	Responsiveness - Willingness to help customers and provide prompt service. 
•	Competence - Possession of the required skills and knowledge to perform the service. 
•	Courtesy - Politeness, respect, consideration, and friendliness of contact personnel. 
•	Credibility - Trustworthiness, believability, honesty of the service provider. 
•	Security - Freedom from danger, risk or doubt. 
•	Access - Approachability and ease of contact. 
•	Communication - Keeping customers informed in language they can understand and listening to them. 
•	Understanding the customer - Making the effort to know customers and their needs. 

It is possible that these criteria could provide an initial framework for the development of quality criteria in other contexts

The ten determinants of service quality interact in the minds of customers with other factors, namely past experience, word of mouth and external communications to create a view of what service is expected.  The diagram (Fig. 2) gives an indication of other factors which might impact on consumer expectations and thereby consumer perceptions of quality. Personal word of mouth communications are still important and still exist in a network environment. Electronic mailing-lists frequently get messages of the type “I’ve looked at this WWW site, and found it useful / not very useful / amusing; here is the URL”. This factor in particular brings another level of subjectivity into the model, and leaves service quality definitions vulnerable to aspects of human behaviour, for example: the desire to emulate other people’s choices associated with and exploited by the fashion industry. This could create potential inefficiencies (Anand et al. 1993).

The work on determinants led to the development of a scale for measuring customer perceptions of service quality called SERVQUAL (Parasuraman et al. 1988; Zeithaml et al. 1990, pp. 175-186; Parasuraman et al. 1991).  This scale has been subject to criticism and refinement and there is a continuing debate about the measurement of service quality and the determinants which should be used (Mathews 1995). 

Fig. 2.	Parasuraman, et al.’s Determinants of Perceived Service Quality

Determinants of service quality									
		Word of Mouth		Personal Needs		Past Experience			
1. Access										
      2. Communication										
3. Competence				Expected					
4. Courtesy				Service					
5. Credibility								Perceived	
6. Reliability								Service	
7. Responsiveness								Quality	
8. Security				Perceived					
9. Tangibles				Service					
10. Understanding the Customer									

Source: Parasuraman, et al. (1985, p. 48)

Companies are constantly encouraged to develop an improved emphasis on service quality.  Schlesinger and Heskett (1991), for example, argue that organisations should abandon the industrial approach to services - the mass-production techniques used in supermarkets, fast-food restaurants and airports - and  adopt a “new model” of service based around customers’ requirements.  Additionally, the service quality debate is connected with the debates on “excellence” initiated by the management guru Tom Peters (Peters and Waterman 1982) and other concepts like market orientation (Caruana and Pitt 1994; Caruana et al. 1994). 
2.3 Total Quality Management
The development of the quality concept in industry has created a requirement for an organisational structure which can include quality concepts at every stage in the planning and delivery of a product or service. The process is called Total Quality Management (TQM). The essence of a TQM strategy is described by Bergman and Klefsjö (1994, p. 22). 

•	Focus on customers 
•	Base decisions on facts 
•	Focus on processes 
•	Improve continuously 
•	Let everybody be committed 

The important insight is that quality becomes a continuous process.  This is especially important for service industries, where customer perceptions of quality are constantly changing.  Quality becomes a process of continuous feedback and improvement.  This process is known as a “quality system”. Bullivant (1994, p. 14) defines TQM as “a commitment to a company-wide culture where everyone is clear of the direction and objectives of the organisation and work in support of each other to achieve these goals”.  Rowley (1996), however, notes that while there is general agreement about the theoretical aspect of TQM, the practical aspects of implementation are more problematic. 
2.4 International standards
National and international standards authorities have turned their attention to quality in recent years.  In the UK the British Standards Institution (BSI) first published BS 5750 in 1979 for quality systems and  the standard was further developed by the International Organisation for Standardisation (ISO) into the ISO 9000 series.

A useful definition of quality is found in ISO 8402: “quality is the totality of features and characteristics of a product or service that bears on its ability to satisfy stated or implied needs” (ISO 8402: 1986).  This has since been refined to “the totality of characteristics of an entity that bear on its ability to satisfy stated and implied needs” (ISO 8402: 1994; EN ISO 8402; BS EN ISO 8402: 1995).

The ISO 9000 approach has been applied in many different organisations and obtaining certification can be essential in some industries (Rothery 1993; Harvard 1994). Although services can be included in the certification process (ISO 9004-2: 1991; EN 29004-2: 1993; BS 5750: Part 8: 1991), ISO 9000 is seen primarily as a product-orientated rather than process-orientated model.
2.5 Benchmarking
Benchmarking is another recent approach to ensuring improvements in quality. Bullivant (1994, p. 1) defines benchmarking as “the continuous process of measuring products, services and practices against leaders, allowing the identification of best practices which will lead to sustained and superior performance”.  Benchmarking can be carried out with relation to different types of organisation (Bendell et al. 1993, pp. 69-70): 

•	Internal benchmarking - comparisons with other parts of the same organisation. 
•	Competitor benchmarking - comparisons with a rival organisation. 
•	Functional benchmarking - comparisons with non-competitive organisations carrying out a similar function 
•	Generic benchmarking - comparison of similar processes in organisations with different functions 

The purpose of the benchmarking process is not just to understand the processes carried out by other organisations, but to enable a considered self-assessment to be made of your own organisation.
3. Database quality
It is only comparatively recently that the library and information community have taken an interest in quality assessment and analysis (Morrison 1994).  In the UK a guide to implementing BS 5750 and ISO 9000 in libraries was published in 1993 (Norton and Ellis 1993) and there has been a increasing interest in quality issues in both Germany and the Nordic countries (IMO 1995, Johannsen 1995).  This interest is largely concerned with issues of quality management (Brophy, Coulling and Melling 1993; Kinnell 1995; Pilling 1996; Seay, Seaman and Cohen 1996; White and Abels 1995) or TQM (Brockman 1992; Martin 1993; Rowley 1996) and has led to the development of a set of IFLA guidelines for performance measurement in academic libraries (Boekhorst 1995) and a draft international standard on library performance indicators (Carbone 1995). In addition to this emphasis, however, there has been an additional focus on defining information quality both with regard to bibliographic records and with electronic databases.  It was the advent of shared computerised cataloguing that made the quality of bibliographic records an issue and studies of this have been carried out both in the UK and USA (Chapman 1994; Thomas1996).

Information quality was not a serious concern for libraries before the advent of electronic databases.  Libraries would select books and journals according to their own criteria, which would usually have some reference to users’ needs or the requirements of the host organisation.  This general approach was later applied to other information formats when they came available, (microforms, video cassettes, LP records, CD-ROMs, etc.), within the financial constraints of the organisation.  Information quality itself began to be a serious concern with the increasing use of electronic databases, both online and on CD-ROM.

Early online databases were regarded with respect by their librarian users.  Basch (1992, p. 85) notes that they were regarded as “nothing short of miraculous” as research tools.  For this reason, it was only by the late 1980s that database users began to make suggestions as to how they could be improved.  The 1989 Annual Retreat of the Southern California Online Users Group (SCOUG) developed a “user wish list” (Basch 1990a), and the following year, a rating list for database quality (Basch 1990c).  The rating list was arranged in a set of ten categories:

1.	Consistency
2.	Coverage and Scope
3.	Timeliness
4.	Error rate/Accuracy
5.	Ease of use
6.	Integration
7.	Output
8.	Documentation
9.	Customer Support and Training
10.	Value to cost ratio
4. The Internet and information quality
SCOUG dedicated their 1995 Annual Retreat to the subject of quality on the Internet (SCOUG 1995).  The participants noted that the Internet was quite different to the database industry as most information providers were not motivated primarily by financial considerations.  Information providers therefore had little or no incentive to improve the quality of their “product”.  SCOUG also noted that whilst certain technical standards were in place, primarily HTML and the work being carried out by the WWW Consortium, the Internet Engineering Task Force (IETF) and commercial organisations like Netscape, there were no content standards.  They considered that people might be prepared to pay for a service which gave access to “charted, safe, quality areas” of the Internet (Ebbinghouse 1995b).  They identified quality issues under the following general headings (Ebbinghouse 1995a):

1.	Credibility
2.	Authority
3.	Indexing
4.	Registration
5.	Reviews / Ratings
6.	Technical
7.	Security and privacy
8.	Feedback / Maintenance / Customer service
9.	Time-size-format pre-alert warnings
10.	Help files
11.	Copyright / Intellectual property
12.	Finding tools
13.	Downloading reliable, transparent and standardised
14.	Online billing
15.	Reliable browsable directory of addresses of sites
16.	Censorship issues, self-censorship
17.	Search engines
18.	Advertising
19.	Payment mechanisms
20.	Able to track usage of your site
21.	Maintenance

Because of the nature of the meeting, the quality issues raised by SCOUG do not all  refer specifically to information quality but to the Internet generally.

An Information Market Observatory report on The quality of electronic information products and services was published in September 1995 (IMO 1995).  It concentrated on quality issues raised by commercial databases, but did make mention of the Internet.  The report identified the main problems as:

1.	Too much information - often redundant and inaccurate,
2.	The lack of centralised control - no editorial function or refereeing.

It also made mention of the World Wide Web (WWW).

“…there is much duplication between sites.  Sites and resources can appear, move or disappear very quickly.  Web sites contain information that ranges from the highly significant through to the trivial and obscene, and because there are no quality controls or any guide to quality, it is difficult for searchers to take information retrieved from the Internet at face value” (IMO 1995).

It concluded that the Internet “will not become a serious tool for professional searchers until the quality issues are resolved”.

Ciolek (1996a, p. 107) has argued that if the WWW is to continue to be of use to scholars and the research community quality issues will have to be confronted:

“Our greatest folly seems to be our willingness to cultivate this global communication system, open to all and sundry, without first ensuring that we have enough useful and trustworthy, accurate and timely information to be circulated across such a networked behemoth”.

Ciolek is the editor of the Information Quality WWW Virtual Library (Ciolek, ed. 1996b) based at the Coombs Computing Unit of the Australian National University in Canberra.  This WWW site gives access to information on quality issues and the Internet, and the unit administers a electronic mailing list called Information-Quality-L which acts as a centre for the world-wide exchange of ideas on Internet quality.
5. Conclusions
Quality is a diverse concept and has been applied to many things.  In management science it is applied to products, services and the process of management itself in TQM.  With information, we are mostly concerned with product quality - ensuring that a product, whether it be a journal, electronic database or WWW page, fulfil an agreed set of criteria.  Additionally, however, information providers might have some interest in service quality with its emphasis on the expectations and requirements of the customer or user and their fulfilment.

Appendix IV: Selection criteria of selective subject gateways
Introduction
There follows short accounts of the selection criteria used by the contacted organisations based on information supplied and on published information.
1. ADAM
A draft document entitled Criteria for including resources was published in April 1996 (Bradshaw 1996a) and describes the criteria and methods used to determine the suitability of resources for inclusion in the ADAM database.  The full guidelines were published later the same year (Bradshaw 1996b)

An important part of ADAM’s selection process is that certain items are eliminated before the quality evaluation process begins.  This includes resources that do not contain any unique information (e.g. only provide links to other resources), resources that have been created by individuals for personal use and those resources which are out-of-date, defunct, inaccurate or superseded.

ADAM’s evaluation criteria are divided into three main areas:
Information content
To what level of detail does the resource go? how superficial / exhaustive is it?
Does the resource contain sufficient basic information, i.e., in a WWW document, contact details, last update details, etc.?
Is the information presented accurately?
Is the information composed well? Is the information within a resource phrased unambiguously?
Is the resource authoritative? who is responsible for the resource? Are they a reputable source? 
Structural design and navigability
Is the information arranged logically?
Is it easy to navigate the resource?
Are hyperlinks ambiguous, i.e., is it obvious where a link is leading you to? 
Are there good back and forward links between pages?
Do you ever find yourself in a position where there are no hyperlinks to anywhere else?
Is the information within a resource arranged consistently?
Is the grammar and spelling accurate?
Are images used effectively or are they over-done?
Is the resource ‘viewable’ effectively (i.e., without loss of essential information and navigability) in non-graphical browsers?
Overall appearance and usability
Is the aim and purpose of the resource obvious at first sight?
Is it attractive/functional?
Does it encourage you to explore further?
Is the balance of links and text good?
Is the balance of text, images and white space good?
How big is the resource?
If a WWW document is long, is it navigable? How long does it take to download?
Are there single document options for those resources that may be printed?
Are there alternative options for those WWW resources which contain Netscape
specific features such as tables?
2. EELS
Selection for EELS is carried out by one of ten to fifteen subject editors, who are usually subject librarians in one of the participating libraries.  To ensure quality, consideration is given to “such factors such as accessibility, maintenance, documentation and reliability of producer information” (Jansson 1996).  There are no firm criteria used or published by EELS, but the guidelines are:
Accessibility 
The resource has to be accessible! There should be no dead links.  Except for some important exceptions, resources should be free of charge.  Commercial databases are only included if they contain rare or otherwise unreachable information.
Documentation and maintenance 
There should be a minimum of information available about the resource itself - who is providing the information? When was it published, last updated, etc.  Editors are supposed to check the resources from time to time, to see whether they are still ‘alive’.
Reliability of producer organisation  
The editors have to use their subject knowledge for this.
Interest
The resource must be of interest to the technical universities involved, from a research or educational point of view.

EELS have not tried to make the criteria more exact. The editors have an occasional meeting to discuss selection criteria questions and to make sure that they follow the same guidelines.
3. EEVL
Selection is currently carried out by EEVL staff and by additional voluntary team members who are all engineering librarians with an interest in the Internet.  Selection criteria are currently under review, but the Project Officer says that when “resources are investigated for inclusion in the database a number of criteria are considered including information content, provenance, authority, usability, durability, reliability of access, and uniqueness within the context of the overall collection. Items which are out of date, inappropriate, strictly local in context or are no longer available are filtered out” (Moffat 1996a).  More detailed criteria can be found in the EEVL team manual (EEVL 1996) which states that the following type of questions should be asked of a resource:

	Does the resource contain substantive information?
	Is the subject matter appropriate for the EEVL target audience?
	Is the resource unique within the context of the total collection?
	Is the information durable in nature?
	Is the information from a reputable source?
	Is the information current?
	Is there any form of quality control?
	Is access reliable?
	Is access free and unrestricted?
	Is there on-line help, or contact details?
	Is there printed documentation?

In practice, resource evaluation will be a combination of many of the above factors, some of which may be in opposition, but the resource has still been considered as a valuable for inclusion.

Content criteria would appear to be the most important on this list, but there is also an interesting interest in the reliability and stability of the information provided.
4. NBW
The selection criteria are published on the NBW Working Home Page (NBW 1996) but can be summarised as follows (information supplied by Marianne Peereboom):
Content:
1. Relevance for the academic or scientific communities
2. Quality of content
Formal:
1. Bibliographic ‘units’ only
2. Full text, multimedia, and referential resources are all included.

These criteria are still being discussed, changed and expanded.
5. OMNI
OMNI have an Assurance Officer (Betsy Anagnostelis) and have an Advisory Group on Evaluation Criteria comprising Alison McNab of the Pilkington Library, Loughborough University and Alison Cooke, a Research Student at the Department of Information and Library Studies, University of Wales, Aberystwyth. (OMNI Consortium 1996a)

In order to ensure the comprehensiveness of the OMNI database, resource descriptions can be created by volunteer helpers who would monitor certain subject areas or inform the OMNI Consortium about important resources.  These volunteers select resources and create resource descriptions for them.  Although the resource descriptions are checked before they are transferred to the OMNI public databases, it is important that the quality selection criteria used is widely available.  Therefore OMNI’s quality selection criteria have been published in a document entitled Evaluating resources for OMNI (OMNI Consortium 1996b)  Fundamentally, OMNI will include a resource if it contains substantive information and is of relevance to the OMNI user community (Ibid.).  The criteria used is broken down into fifteen sections:

	Scope
	Audience
	Authority
	Provenance
	Accuracy of information content
	Uniqueness / comparison with other sources
	Currency / frequency and regularity of updating
	Accessibility and usability
	Charging policy
	Special requirements
	Software reliability
	Copyright
	Language
	Design and layout / user interface
	User support / documentation

The main emphasis is on content criteria, with some importance being given to design and ease of use issues.  “... [OMNI] are primarily interested in the value of a resource in terms of information content; quality of design or appearance are of secondary interest, even though they may affect the overall usefulness of a resource” (Ibid.).  OMNI also comment that although the evaluation process will take in a combination of the criteria listed above, the important thing is the assessor’s “overall impression about the value of a resource to the OMNI user community” (Ibid.).
6. RUDI
The RUDI project intends to build up a collection of hypermedia materials on urban design - mostly stored on its own server.  Selected resources from other sites will be included on the service.  Selection will be carried out by team members (subject librarians) at Oxford Brookes University in collaboration with the RUDI Internal Advisory Group and Steering Committee.

As of the end of July 1996 selection criteria were still in the process of being formulated.
7. SOSIG
SOSIG only selects resources perceived to be of quality.  An e-mail cited on the SOSIG home page states that “Given the amount of information on the net, the real value of a resource such as yours [SOSIG] is, paradoxically, not that it is comprehensive but that it is selective of high quality resources” (SOSIG 1996).  The same document states that SOSIG filters out resources that are of little or no use to our users. This process also weeds out material that is out of date, inappropriate, strictly local in context or refers to resources that are no longer available” (Ibid.).
Relevance
Is it relevant to the subject area, user profile (i.e. education and research)
What is the scope of the resource (geographical limitations, etc.)?
Is the information substantive - a resource which consists of a collection of links to other resources will normally not be included unless there is substantial annotation or value-added information.

Note - UK academic departmental pages are included in the database (as researchers often want to make contact with other departments) but similar pages from other countries are not deemed suitable for inclusion.
Features
Is the information accurate, comprehensive? (often hard to validate this information)
Is there any on-line help/information?
Perceived value
Is it from an authoritative source?
Reputation of information provider?
Is the information peer reviewed?
Uniqueness
Is the resource available elsewhere (different formats, sites)?
Is there similar/better subject material available?
Does the material have any relation to other works?
Is it an often cited source?
Maintenance
Is the information being maintained/updated?
Is the information being maintained/updated?
Is the information provider likely to be able to maintain the information (unlikely in the case of information provided by students)
Meaning over time
Does the information have a time limit to its usefulness i.e. timetables, schedules, conference announcements etc.?
Presentation of information
Is it presented well, easy to use and manage?
Physical access
Are the connections to the site providing the information reliable and stable?

Appendix V: Selection criteria of other Internet services
1 Introduction
Nineteen  selective Internet services were analysed with regard to  the criteria they use to select documents for their database. Twelve of these are general services, four are regional or national services and three are subject specific services.
General services
•	Argus Clearinghouse
•	WWW Virtual Library
•	Cyberhound
•	excite NetDirectory
•	Galaxy (no information available)
•	Infoseek select sites (no information available)
•	Lycos / Point Communications, Top 5% reviews
•	Lycos / A2Z 
•	Magellan’s Reviews 
•	NetFirst 
•	Webcrawler select 
•	Yahoo (no information available)
National, regional (selection)
•	Jubii (Denmark) 
•	DINO (Germany) 
•	SUNET (Sweden) 
•	UK Web Library (UK)
Subject specific
•	City.Net 
•	GeoSurver (no information available)
•	Ei Village (no information available)
Argus Clearinghouse

source: <URL: http://www.clearinghouse.net/docs/submit.html> [Accessed: 19 July 1996]

Selection criteria

•	resource must be accessible via the Internet
•	“Guides must point to other Internet information resources ... which deal with one specific topic (or more than one if they are related to each other)”
•	“In order to create some consistency and make things easy for end-users, we are requiring that the following information be displayed prominently within each guide: 
	
1.	author’s full name 
2.	author’s title and institution/company 
3.	author’s email address 
4.	brief biography of author (recommended--it’s good practice to let users know who you are and why they should trust your perspective) 
5.	date of last update to guide (older guides eventually will be moved to less prominent locations within the Clearinghouse) 
6.	version number for guide (recommended)” 

•	Guides must be free of charge for end users. 

Rating criteria

source: <URL: http://www.clearinghouse.net/docs/ratings.html> [Accessed: 19 July 1996]

•	“Level of Resource Evaluation: Evaluative information provides users with a subjective sense of the quality of the Internet resources, including: 
	
•	Quality of the content of resources (e.g. discussion in mailing lists and Usenet newsgroups, information in a Web site). ... 
•	Authority (e.g., reliability) of resource authors. “ 
	
•	“Level of Resource Description: Descriptive information provides users with an objective sense of what the Internet resources cover, including: 

•	Description of the resources’ content (ranging from keywords to abstracts). 
•	Description of the traffic levels, level of moderation, features (e.g., digests) for mailing lists and Usenet newsgroups. 
•	Intended audience for the resources. 
•	Description of the update frequencies for resources 
•	Access instructions for the resources. 
•	Technical performance levels of the resources (i.e., a server is frequently down). “ 
	
•	Guide Organisational Schemes (How are the resources in the Guides organised?)
•	“Assessment of the resources’ usability (e.g., document layout, readability, appropriate use of graphics, organisation)”
•	Meta information

•	“Guide Design: Quality guides balance aesthetics with usability: 

•	Images (are images attractive, do they support ease of navigation, do they load quickly). 
•	Layout (does the author make appropriate use of headers, mixed font sizes, and white space). 
•	Navigational aids (it is easy to find your way around, do you have a consistent sense of context or understanding of where you are in the guide at any given time). “
City.Net (excite)

source: <URL: http://www.city.net/cnx/credits.html> (sources for their own searches)

<URL: http://www.city.net/cnx/cnx_faq.html> [Accessed: 19 July 1996]

•	city pages (geographic and tourist information, maps etc.)
•	 (“ We add cities when they have been submitted to us or when we find them through our own searches on the Net. “)
Cyberhound

Rating criteria

source: <URL: http://www.thomson.com/cyberhound/ratings.html> [Accessed: 19 July 1996]

comment: They have an overall rating and individual ratings for content, design and technical merit.

•	“Has the data been updated recently? “
•	“Authoritative/reliable information: Who’s providing this information? Do you trust them?”
•	“Comprehensiveness: Does the site cover all the facets of a particular topic? Is everything you expect to find in the site there? Are there any logical gaps of information? “
•	“Unique or hard-to-find information: Does the site contain information that is not readily available? Would you have to look long and hard to find similar data, or is it a dime a dozen?”
•	“Is this a popular topic that people will want to know about?” 
•	rating for design
DINO

source: <URL: http://www.dino-online.de/dino.html> [Accessed: 19 July 1996]

•	sites in German (sites on German, Austrian and Swiss-German servers)
•	probably only sites submitted by users

Excite NetDirectory

source: <URL: http://www.excite.com/Handbook/NetReviews/index.html> [Accessed: 19 July 1996]

•	“... rather than pointing you at particular Web pages and Usenet articles, NetReviews points you to Web sites and Usenet groups”
Galaxy

source: <URL: http://galaxy.einet.net/about.html> [Accessed: 19 July 1996]

•	“Our index includes only those pages actually submitted to us.”

Lycos Point Top 5%

source: <URL: http://point.lycos.com/faq/#choose> [Accessed: 19 July 1996]

•	“best, smartest, and most entertaining sites”
•	“Point’s staff of reviewers surfs the Web daily looking for the best, smartest, and most entertaining sites around (We also look at suggestions we receive in our submit box). If we review a page, it means we think it is among the best 5% of all Web sites in content, presentation, and/or experience. Some sites are tops in all three categories; others qualify due to fabulous content or exceptional presentation. And some make it in just because they’re so deliciously silly. 
•	“Point makes no distinction between commercial, private, or student pages. Excellence is our only criterion.” 

Rating criteria

source: <URL: http://point.lycos.com/faq/#ratings> [Accessed: 19 July 1996]

•	“Up-to-date? “
•	“Is it accurate?”
•	“Just how broad, deep, and amazingly thorough is the information? Are there good links?... Complete?”
•	“Does it lead visitors through the information nicely? Does it use video, audio, and original graphics? Does it break new ground?”
•	“Experience: This is the key rating. Is this fun? Is it worth the time? Will we recommend it to friends?”
•	“Presentation: Is the page beautiful? Colorful? Easy to use?”
Jubii

source: email from “Jakob Faarvang” [Accessed: 18 July 1996]

•	language: “The site has to be Danish”
•	“If a page is almost empty or without content, we do not accept it.”
Lycos / A2Z

source: <URL: http://a2z.lycos.com/faq.html> [Accessed: 19 July 1996]

•	“the most linked-to (and therefore the most rewarding) sites on the Net”
•	“Is the site sponsored by a company, organization or individual widely recognized as an authority or expert in the category field?
•	“Does the site contain valuable information which is not overly duplicated by related sites already existing in the directory?”
•	“Does inclusion of the site enhance or complement existing sites in a given category?”
•	“Does it contain breakthrough content or design elements?”
Magellan’s Reviews

Selection criteria

source: <URL: http://www.mckinley.com/feature.cgi?faq_bd> [Accessed: 19 July 1996]

•	“Sites will be selected for review based on the usefulness and entertaining quality of their content,”
•	Exclusion of sites relating to pornography, paedophilia, or hate groups

source: email from <temp2@mckinley.com>  [Accessed: 22 July 1996]:

•	“... if we have a few sites on that subject, then we would look for sites that would expand that area, rather than just repeat what all the other sites have.”

Rating criteria

•	“Depth: Is it comprehensive and up-to-date?” 
•	“Ease of Exploration: Is it well-organized and easy to navigate?”
•	“Net Appeal: Is it innovative? Does it appeal to the eye or the ear? Is it funny? Is it hot, hip, or cool? Is it thought-provoking? Does it offer new technology or a new way of using technology?”
NetFirst

source: <URL: http://www.oclc.org/oclc/netfirst/faq.htm> [Accessed: 19 July 1996]

Concerning a site: 

•	“provided there is sufficient information with which to create a descriptive record”

Concerning lower level objects: 

•	“either proven to be, or expected to be, durable”
•	“expected to be of general interest to our users”
•	“substantial, where size is a primary determinant”
•	“ the quality of the data is the primary criterion”
SUNET

source: <URL: http://www.sunet.se/sweden/help-sv.html> [Accessed: 19 July 1996]

•	Swedish resources which are of interest to more than one person
•	most of the resources submitted by users
UK Web Library

source: <URL: http://www.scit.wlv.ac.uk/wwlib/misses.html> [Accessed: 19 July 1996]

•	negative selection: sites about nudes, bomb making, pornography probably won’t be included in the catalogue.
Webcrawler select (GNN select)

source: <URL: http://www.gnn.com/gnn/wic/wics/support/about.rescat.html> [Accessed: 19 July 1996]

•	“sites that receive frequent updates”
•	“Our aim is to collect the best resources within each of the subject areas covered by the catalog”
•	“effectively designed, demonstrate a feel for Web culture”
•	“available to users at no cost”
WWW Virtual Library

general source: <URL: http://coombs.anu.edu.au/SpecialProj/QLTY/QltyLinks.html> [Accessed: 29 Nov. 1996]

The WWW Virtual Library Project is a distributed project - the replies below have been split according to the maintainer of the system.

source: T.Matthew Ciolek <tmciolek@coombs.anu.edu.au> (WWW VL System:  Aboriginal Studies 
 Asian Studies, Buddhist Studies, Information Quality, Social Sciences, Tibetan Studies)

Selection criteria

•	If there is a lack of information in given area - any online site will do. Later, these links are replaced with ones leading to better & richer sites 
•	If a site provides factual information (= mere opinions, literature, ideologies etc. are out) 
•	If a site provides original information (= does nor replicate information available elsewhere) 
•	If a site provides with their own information are preferred to sites and links to other sites 
•	If it is a WWW system although gopher or ftp sites are also considered 
•	If the information is well organised and presented 
•	If the information is attributable to an author and/or editor. Anonymous, unsigned, unattributed information is not to be trusted. 
•	If the link is reasonably fast and stable 
•	If the link is sufficiently focused on an interesting resource, links to general, introductory pages are replaced with links leading directly to the ‘heart’ & ‘marrow’ 
•	If the site is frequently updated, improved and enlarged (in other words, I tend to establish links to electronic equivalents of research papers, books and encyclopaedias - as opposed to mere posters, postcards and poststamps)

source: Nelson C. Baker <nb2@eiffel.ce.gatech.edu> (WWW VL System: Civil Engineering)

Selection criteria: 

•	I only include other URL’s of sites where the maintainer has given me permission to list the site. 
•	I browse the site to make sure that it is viewable and contains information related to my VL

source: Thomas Hartley <thartley@uconnvm.uconn.edu> (WWW VL System: Political Science) 

Selection criteria: 

•	So far, we have included everyone who requested inclusion. 
•	When the page becomes too large to proceed in this way, I plan to create (or try to find other people to create) separate but related pages.

source: CyberWeb <web@sowebo.charm.net> (WWW VL System: World-Wide Web Development)

Selection criteria: 

•	Relevance, Accessibility, and Quality. 
•	I occasionally get requests to link to flower shops/record stores/etc., which I ignore. They have evidently never even visited my site. 
•	I visit all sites before linking them. If I can’t access them then they won’t get linked. 
•	I add any introductory paragraph to my URLs database so that annotation about the site is available. 
•	I try not to be too judgmental about site quality if it has something useful to offer; but just once or twice the presentation was too bad. 
•	I get most sites myself through newsgroup, mailing lists and surfing but I plan on doing less of that as the volume of link requests increases.

source: Jonathan Bowen <Jonathan.Bowen@comlab.oxford.ac.uk> (WWW VL System: Museums, Publishers)

Selection Criteria: 

•	Whether a proper URL is supplied. 
•	Whether the URL works! 
•	Whether the material is relevant. 
•	How well the material is organised. 
•	How good the links are to further relevant information. 
•	How well laid out the pages are. 
•	How fast the page loading is. 
•	How considerate and well thought out the use of graphics is. 
•	Whether I have heard of the organisation previously (This may speed my response if so.) 
•	Whether I have the time to spare!!

source: <valiant@mordor.com> (WWW VL System: Automotive)

Selection Criteria: 

•	I use anything that I can find which I have verified as being current and working. 
•	I would like someday to switch to non-commercial, but in this area, that’s probably hard to separate. Maybe I will sooner or later switch to a core view --
•	automotive rather than auto accessories.

source: John Lock <jlock@mindspring.com> (WWW VL System:  Beer & Brewing)

Selection Criteria: 

•	I try to only include links to pages that contribute something of substance to the overall topic. I get a lot of requests for listing pages that are nothing more than a
•	collection of links to other pages, most of which are already in the library. I prefer links that provide some useful content. 
•	Commercial or advertising pages are placed in their own category. Related to that, I avoid pages that “appear” to represent a company, but are merely published
•	by an individual who likes their product(s). 
•	I reject all pages that encourage alcohol abuse. 
•	I don’t list pages that represent a conflict of interest. For example, I was recently requested to list a page for a small branch of a larger organisation which does
•	not have a WWW presence. The “head office” e-mailed me requesting the branch office page not be listed for two reasons: 
•	The head office was in the process of establishing their WWW presence and wanted to have editorial control of branch pages. 
•	The branch page did not present the appropriate image that the head office was trying to maintain. 
(I e-mailed back stating that I would not list the page until the two parties could come to an agreement)

source: Jesper Weissglas <jesper.weissglas@mikrolit.se> (WWW VL System: Furniture & Interior Design)

Selection criteria: 

•	I look at the suggested site, and anything that I find matches my subject gets added... 
•	I have divided the sites into Academic, Commercial and a few others...

source: Thomas Stein <stein@wiz.uni-kassel.de> (WWW VL System: Irrigation)

Selection criteria: 

•	I try to include all sources of real information (on-line information) excluding pages which just keep information like addresses to write to or so. Unfortunately it is not always possible to stick to that as relatively little irrigation information is available on the net. 

Appendix VI: Selection criteria found via the literature review
The literature review identified sources that included quality selection criteria for Internet services and/or traditional library services.  Selection criteria were drawn from the following sources:
On-line resources
Caywood, C., 1995, Library selection criteria for WWW resources. 
<URL: http://duckdock.acic.com/carolyn/criteria.htm> [Accessed:. 24 Apr. 1996]

Ciolek, T.M., (ed.), 1995, Criteria used to select links for resources’ catalogues. 
<URL: http://coombs.anu.edu.au/SpecialProj/QLTY/QltyLinks.html> [Accessed: 11 Aug. 1996].

Ciolek, T.M., 1996, The six quests for the electronic grail: current approaches to information quality in WWW resources. Revue Informatique et Statistique dans les Sciences Humaines (RISSH) (Centre Informatique de Philosophie et Lettres, Université de Liège), No. 1-4, 1996. 
<URL: http://coombs.anu.edu.au/SpecialProj/QLTY/TMC/QuestMain.html> [Accessed: 26 June 1996]

Cornell University Library, 1996, How to critically analyze information sources.
<URL: http://urislib.library.cornell.edu/skill26.html> [Accessed: 31 May 1996]

Cox, M.D, 1996, LogLink Logistics: web page review criteria. 
<URL: http://www.commerce2000.com/logistics/criteria.html-ssi> [Accessed: 30 Apr. 1996]

Fisher, T., 1995, Evaluation of an Internet information resource. 
<URL: http://www.vuw.ac.nz/~thelma14/eval.html> [Accessed: 30 Apr. 1996]

Grassian, E., 1996, Thinking critically about World Wide Web resources. 
<URL: http://www.ucla.edu/campus/computing/bruinonline/trainers/critical.html> [Accessed: 22 Apr. 1996]

Information Market Observatory (IMO), 1995, The quality of electronic information products and services. IMO Working Paper 95/4. Luxembourg: IMO, September. 
<URL: http://www2.echo.lu/impact/imo/9504.html> [Accessed: 30 Mar. 1996]

Janicke Hinchliffe, L., 1996, Resource selection and information evaluation. 
<URL: http://alexia.lis.uiuc.edu/~janicke/Evaluate.html> [Accessed: 29 Apr. 1996]

McLachlan, K., 1996, WWW CyberGuide ratings for content evaluation. 
<URL: http://www.cyberbee.com/guide1.html>. [Accessed: Sep. 1996]

Rettig, J., 1995,  Putting the squeeze on the information firehose: the need for ‘Neteditors and ‘Netreviewers.  
<URL: http://www.swem.wm.edu/firehose.html> [Accessed: 29 Nov. 1996]

Smith, A., 1996a, Criteria for evaluation of Internet information resources.
<URL: http://www.vuw.ac.nz/~agsmith/evaln/> [Accessed: 11 Apr. 1996]

SOFWeb, Research and the Internet.
<URL: http://www.dse.vic.gov.au/netiget2.htm> [Accessed: 30 Apr. 1996]

Stepno, R. and Henshaw, R., 1995, Quality of information … and disinformation online. 
<URL: http://blake.oit.unc.edu/~rbstepno/disinfo.html> [Accessed: 31 May 1996]

Sun Microsystems, 1995, Guide to Web style: quality. 
<URL: http://www.sun.com/styleguide/tables/Quality.html> [Accessed: 3 Sep.1996]

Tillman, H.N., 1996, Evaluating quality on the Net. 
<URL: http://www.tiac.net/users/hope/findqual.html> [Accessed: 24 Apr. 1996].

Victoria University of Wellington, Department of Library and Information Studies, 1995b, Criteria for evaluation of Internet information resources. 
<URL: http://www.vuw.ac.nz/dlis/courses/847/m2resevl.html> [Accessed: 29 Nov. 1996]
Print resources
Collins, B.R., 1996, Beyond cruising: reviewing. Library Journal, Vol. 121, no. 3, February 15, pp. 122-124.

Cooke, A., et al., undated, Developing a technique for assessing the quality of networked information. Preliminary results taken from an unpublished progress report. Aberystwyth: University of Wales Aberystwyth, Department of Information and Library Studies.

Farber, M., 1994, The quality of information on Internet. Computer Networks and ISDN Systems, Vol. 26, Suppl. 2, pp. S75-S78.

Pratt, G.F., Flannery, P. and Perkins, C.L.D., 1996, Guidelines for Internet resource selection. College and Research Libraries News, Vol. 57, no. 3, March, pp. 134-135.

Stoker, D. and Cooke, A., 1993, Evaluation of networked information sources. In: Helal, A.H. and Weiss, J.H. (eds.), Opportunity 2000: understanding and serving users in an electronic library: 15th International Essen Symposium … Festschrift in honour of Herbert S. White. Publications of Essen University Library, 15. Essen: Universitätsbibliothek Essen, pp. 287-312.
Selection criteria from the literature review
A list of the selection criteria found via the literature review is given below.  Each and every criteria was noted and is given below as raw data, i.e. as it was before any categorisation took place.

What is the coverage in the collection?

Is there any factual information?

Is there any original information?

Must be WWW site

Is it well organised?

Is it well presented?

Information must be attributable to an author or editor?

Link must be reasonably fast

Link must be reasonably stable

Frequently improved and enlarged and updated?

Maintainer has given permission for site to be included?

Must visit the site before linking to it

Where did you learn of the site - from a respected source?

Are the external links to further information relevant?

Have you heard of the organisation/author/editor previously?

Is the site non commercial? 

Advertising present? 

Is the page what it appears to be - by who it says it is by?

Does the URL support the claim of authorship? 

Is the URL a university server?

Contains more than just contact details

Contains more than just list of links

Is the site still useful with an ASCII browser like LYNX?

Is it written in standard HTML?

Have proprietary extensions to the HTML been added that some browsers will not recognise?

Does it use standard multimedia formats?

Do parts of it take to long to load?

Is it usually possible to reach the site or is it overloaded? 

Has the URL changed? 

Are you being redirected to a new URL? (stability) 

Is it open to anyone on the Net or do parts of it require fees?

Are any rules for use stated up front?

Are the individual Web pages concise or do you have to scroll forever?

Do essential instructions appear before links and interactive portions?

Do all the parts work?

Is using the site intuitive, or are parts likely to be misunderstood?

Can you locate a particular page from any other page?

Does it look and feel friendly?

Does it do more than can be done with print?

Can the user interact with it?

Are interactions secured if they involve private information? 

Is the scope clearly stated?

Are headings clear and descriptive?

Is the title informative?

Are the limits stated?

Does it include jargon?

Is it organized by the needs of the user?

Does the content fit the scope?

Are the content and links clearly described and suitable to the expected audience?

Is the content amplified over time or is one visit all it’s worth?

Is the origin of the content documented?

Is the content of the document verifiable?

Is the amount of bandwidth commensurate with the usefulness of the content, or are the graphics or multimedia included simply to show off?

Is the site sponsored?

Is the publisher a recognised authority?

Does the page provide other sources which could be contacted for confirmation?

Does the page provide other sources which could be contacted for further information?

Is there an Email address for the author?

is there an Email address for the publisher?

Is there a common link to the page from a recognised authority?

Does the page cite a bibliography or provide references to confirm the accuracy of the information’

Has the site been reviewed by a ‘content’ reviewing agency?

Consistency

Error-rate accuracy

Customer support and training

Timeliness

How current is the material included in each update?

Are time-sensitive sources available in near real-time?

What data sources are used?

Is there a prevalence of typographical errors?

Is some of the information incomplete?

What quality control procedures does the producer employ?

Does the system require special software?

Can the user sign up on-line or must written contracts be negotiated first?

Does the system have a search facility?

Can pages or portions of a document be printed separately?

Is on-line documentation timely, accurate and readable?

Is print documentation available?

Is the documentation searchable?

Does the information overlap with items already in the collection - does it duplicate?

Do you know the educational background of the author?

Do you know the occupational background of the author?

Is the author listed on say, the campus directory/organisational directory?

Is there an index?

Is the resource indexed electronically? (Is the index kept up to date?)

Demonstrated importance through availability: linked by multiple Internet sites

Demonstrated importance through availability: available in multiple formats (print/CD-ROM etc.)

Are the downtimes infrequent and announced?

Dan you review the peak usage time for the resource? (to assess reliability)

Is the site mirrored?

Is specific hardware needed?

Does the source avoid ‘blink’ features?

Are all the pages dated with the last revision date?

Are all the links clearly labelled?

Do all the links serve an easily identified purpose?

Are graphics/sounds/videos/ clearly labelled and identified?

Do the graphics/sounds/videos serve a clear purpose appropriate for the intended audience?

Is the content of linked sites appropriate for the intended audience?

Is the content free of bias - or is the bias made clear to the audience?

Are large graphics hindering a fast download time?

Is there a sufficient number of links to guide you through the document, and to allow you to retrace your steps?

Is the scope stated or implied?

Breadth - are all aspects of the subject covered?

Depth - to what detail does the subject go?

Time- is the information in the resource limited to certain time periods?

Are sources of the information stated? 

Is the resource pitched at the right level for the intended audience?

Does the resources fulfil the stated purpose?

Does your user group have the connectivity to access this resource?

If it is a static resource (not updated) will the information be of lasting use to the audience?

Do the stated dates respond to the information in the resource?

Does the organisation or person hosting the resource seem to have the commitment to the ongoing maintenance and stability of the resource?

Uniqueness - is the resource available in other forms e.g. other sites, print, CD-ROM,?

If the resource has been derived from another format e.g. print, 

How effectively can information be retrieved from the resource?

Is a useful search engine provided?

Has a resource been designed to work well with one interface (e.g. Netscape), but be difficult to use with others (e.g. Lynx)?

Why is the information there?  What was the motivation of the information provider when they made the information available?  

Cross-checking the information - can you do this?

What is the purpose of the resource?

Relative value of the resource in comparison with others available on this topic?

Relative value of the resource in comparison with others already in the collection?

What other resources are available in this area?

What are the link selection criteria if any?

Is the site inward focused i.e. not just linking to other, external sites?

Is there an appropriate balance of inward pointing links and outward pointing links?

When was the web item produced?

When was the web item mounted?

Are there references to sites that have moved?

Does the document follow good graphic design principles?

Do the graphics and art serve a function or are they decorative?

Do the icons clearly represent was is intended?

Does the text follow basic rules of grammar, spelling and literary composition?

Is there an element of creativity, and does it ad  or detract from the document itself?

Is there an option for multimedia browsers/line-mode browsers?

Is attention paid to the needs of the disabled, e.g. large print and graphic options; audio?

Is appropriate interactivity available?

When it is necessary to send confidential information out over the Internet, is encryption (i.e. a secure coding system) available?  How secure is it?

Has the author had previous publications of note?

Is the information written on a topic in the author’s area?

Can information located in the publication itself be used to determine the author’s credentials?

Is the author someone who has been cited frequently by respectable sources?

Is the publisher  ‘University press’?

Is the information scholarly rather than popular?

Does the information appear to be valid, and well researched?

Is it primary material?

Is it merely vanity publishing?

Has the material been disseminated by a trade publisher?

Are there stated criteria for inclusion of information?

Appropriateness of format

Is there an opening mission statement/statement of the purpose of the resource?

Does the site give, or take?

Is original work available at this site?

What type of resources of relevance to the audience: listings, indexes, data, abstracts, addresses, phone numbers, reports, studies, summaries, reviews, surveys, dictionaries, definitions, news, newsletters, stock quotas, annual reports, service descriptions, regulations, technical data, guides, historical information?

What technology is appropriate for the audience: forms, ismaps, searches, databases, cgi scripts, Java applications, frames, web sites, gopher site, ftp, WAIS, telnet, effective application,

If the site contains data or information that is time-sensitive how current is this data and information?

How time sensitive is the information, and how does this relate to frequency of update? 

Criteria balance

What type of site is relevant to the audience: academic, government, commercial, trade/industry, non-profit, private?

Information from which geographical areas is relevant to the audience : local, nation-wide, from particular countries or continents or world wide?

Quality control: publishers/peer review/refereed journals/ 

Is the publisher known to you and reputable?

Have you heard of the author before?

Appendix VII: User surveys
For each service a description of the survey aims, methods and response is given, followed by a summary of the findings, if appropriate,  in three critical areas:

•	User requirements
•	User behaviour
•	User satisfaction

A number of points about these user surveys should be borne in mind:

•	The response rates vary from over 200 to less than 30.  The level of response should be noted carefully when considering the findings.
•	The aims of the surveys vary: some were designed to feed into the development of the service; some were done as part of the evaluation process; and some were not directly associated with the service but to a related project.
•	The different aims of the surveys means that some surveys did not directly cover the user-factors related to the development of resource selection criteria.
•	The subject gateways vary considerably in their stages of development.
1. ADAM
In the early part of 1996 ADAM conducted a questionnaire survey of users and potential users via the ADAM server and an email shot of a number of discussion lists.  Results are due in Autumn 1996 (to be examined in a later stage of DESIRE).
2. EELS
User behaviour
Some statistics taken from the period between the 12th of March 1995 and the 18th of June 1995 show that the majority of users browse the system for resources as opposed to searching (32,000 accesses browsing, 3,500 searching).  Although it is suggested that probably the majority of the browsing accesses are in fact users finding pages on the system from outside the service itself i.e. they are coming from WWW search engines.  The main usage by domain/country is from Sweden (between 50-75%)
3. EEVL
In April/May 1996 EEVL surveyed a selected sample of users from 6 pilot sites (UK universities).  The survey was conducted via workshops held at the sites, with three stages to the data collection:

•	A background questionnaire (before having used EEVL)
•	A group discussion
•	An initial impressions questionnaire (after having used EEVL)

The aim of the workshops was to introduce the EEVL system , and to carry out evaluation of the pilot phase of the project.  Responses were collected from 81 users.
User requirements
The survey elicited a list of the types of information that users were searching for on the Internet:

1.	Users wanted to find information on other people/departments doing similar research
2.	Full text of reports/papers
3.	Company information -particularly information on products
4.	Software (FTP sites for downloadable software)
5.	Engineering education
6.	Conferences
7.	Search tools like BIDS
8.	On-line journals
9.	FAQs
10.	IEEE abstracts
11.	Technical data i.e. electronics datasheets
12. Physical property data

-  In general participants were supportive of not including newsgroups in the database.
User behaviour
a)  Behaviour when using the Internet generally:
•	The majority of users said they used computers every day.
•	Respondents felt they were experienced users of email and word-processors, but less experienced users of FTP and gophers.  Most said they were of ‘average’ experience at on-line searching.
•	Despite wanting to find information on the Internet, users often refrained from searching because they felt ‘the costs of locating the information outweigh the usefulness of the information itself’.  Users found searching slow because of the slow connections and the unfocused nature of the  ‘hits’ from search engines.

b)  Behaviour when using EEVL:
•	Users found the searching and browsing facilities easy to use.
•	They tended to use the search facility more than browse, although they used the browsing to get a feel for the range and depth of material in the database.
User satisfaction
•	The initial reaction to the EEVL service was ‘overwhelmingly positive’.  
•	Users commented that they liked the inherent simplicity of the system.
•	They expressed a desire to access more information via the Internet and recognised that EEVL could save them time in doing so.
•	A service which focused on quality resources in engineering was welcomed.
•	Two concerns were raised over the size of the database (small), and the future of the service after the end of the two year project.
4. OMNI 
An on-line user survey was conducted as part of the ROADS evaluation process in June 1996.  The aim was to gauge the users of the ROADS system.  A questionnaire was placed on the OMNI server for a limited period.  There were 23 useable returns, with 16 UK universities represented.
User requirements
Users were asked what other services they would like to see on OMNI and what type of resources they would like to find.  The answers included:

•	‘Ability to choose purely British sources’
•	‘More of the same, but in different subject areas’
•	‘increased coverage’ (generally and in specific subject areas)
•	Respondents worked in a wide range of different disciplines suggesting a breadth of coverage is needed.
User satisfaction
Things that users thought were particularly good about OMNI:

•	‘the fact that sources had been evaluated, and the resulting quality information ‘(the quality control)
•	the range and types of resources available
•	the well organised structure

Things that users thought were not particularly good about OMNI:

•	the small number of resources available on the service

In general users’ comments suggested that users are pleased with what the service is providing and wish for more of the same.
5. RUDI
A paper questionnaire was sent out in February to all UK Higher Education institutions (62) which are known to have teaching and/or involvement in one or more areas of urban design, as well as an enclosure in Urban Design Quarterly to reach professional bodies and practitioners. (The main aims of the survey were to:

•	define the requirements of RUDI by end-users
•	provide a basis to develop a strategy for selecting and organising material on RUDI

This report is based on the provisional results of the survey.
User requirements
RUDI differs somewhat from most of the subject gateways in their intention to host primary material on their server, in addition to a gateway to resources available elsewhere.  Many of the questions were to establish the type of material users would like to see on the service.  

•	There was a marked preference for UK based information although material on Europe and North America was also required.  
•	Most respondents would like to have the facility of an electronic noticeboard.
•	Respondents were asked to specify the likely end use of material on RUDI, most academics indicated teaching, closely followed by research.  Commercial practitioners however would use it to develop their own design ideas.
User behaviour
The project is still in its preliminary stages and only publicity material is available at this stage.  However with regard to usage, results from the preliminary survey suggest that current usage of the Internet amongst design schools is relatively high and anticipated use of the RUDI service is also high.  Almost all of the academic respondents said they were likely to use the system as well as a majority of the practitioners.  
6. SOSIG

There have been two user surveys which relate to SOSIG:

•	Evaluation forms/anecdotal evidence collected via Internet workshops prior to the establishment of SOSIG (1992-4).

•	An on-line survey done as part of the evaluation of the ROADS project (1996).

The survey preceding SOSIG
Over 200 evaluation forms and a wealth of anecdotal evidence were collected from users by Nicky Ferguson when he conducted Internet workshops in UK universities between 1992 and 1994, prior to the SOSIG project.  Working for the ESRC, Ferguson conducted workshops entitled ‘The Internet for Social Scientists’ and ‘Exploring the Networks and Accessing Information’.  His observations of users and the feedback he obtained from them are summarised below.
User requirements
•	Users want a simple way of finding Internet resources that can help them with their work.
•	Users want to be able to find resources of a high quality without having to spend lots of time browsing the Internet.
•	Users want a ‘one-stop-shop’ for social science resources so that they can locate resources of a high quality, quickly and easily.
•	Internet resources need to be accessible to users with any kind of network connection and any kind of hardware.
User behaviour
•	Anecdotal evidence from Nicky Ferguson suggests that users tended to find it easy to browse the Internet but difficult to find anything of actual use to their work.  Browsing for useful resources was not proving to be an effective use of their time.
•	The workshop evaluation sheets suggest that some users did not feel confident that they were finding the Internet resources most appropriate to their work.  They lacked awareness of the range and number of resources available to them via the Internet and lacked the confidence to use the Internet in the absence of the trainer.
User satisfaction
This survey preceded SOSIG so user satisfaction could not be measured.
The 1996 survey of SOSIG users.
An on-line questionnaire was mounted on the SOSIG server for two short periods in 1996 (For 3 days in January and for a week in July) as part of the ROADS evaluation process.  The aim was to gauge the users of the ROADS system.  Users were invited to fill in the questionnaire on a voluntary basis, and there were 40 useable returns.
User requirements
Users were asked what other services they would like to see on SOSIG and what type of resources they would like to find.  The answers included:

•	‘more of the same high quality material.’
•	‘indexes kept in frames while browsing/searching’
•	‘more regular updates’
•	A number of requests were made for more resources to be available in particular subject areas
•	In terms of the types of resources that users wanted to find on SOSIG, the survey found that users wanted ‘more of the same’.  Users want more of the same high quality material.
•	Suggestions for improvements included: ‘more international resources’; ‘a social science software site’

The respondents were from a wide range of disciplines within the social sciences, suggesting a breadth of coverage is required.
User behaviour
•	Most of the users used SOSIG for their own use, but 10 users were carrying out searches for other people.
•	Most users were using SOSIG for research purposes (26/40), as opposed to teaching or learning purposes.
•	Generally users liked having both browse and search facilities, and found them both easy to use.
User satisfaction
Things that people thought were particularly good about SOSIG:
•	‘quality of resources’
•	‘discipline based structure’
•	‘ease of use and evaluation of sources’
•	‘links to other useful sites’
•	‘URL descriptions’
•	‘clear structure’
•	‘range of coverage’
•	‘tree structure’
•	‘appearance’

In terms of the quality of the information 31/40 users felt the resources found on SOSIG were either ‘good’ or ‘very good’.  No users found the information to be of a ‘poor’ quality.

Things that people thought were not particularly good about SOSIG:
•	‘Not many links yet’
•	‘not updated enough’
•	‘sometimes difficult to find certain links listed’
•	‘need to turn back pages to find resources on the same index’
•	‘finding by author’s name’
•	‘no indication of the number of hits from a search’
•	‘poor description of what the SOSIG database offers’
Conclusions
As noted in the introduction most of the surveys were not designed to elicit feedback related to resource selection criteria (although OMNI have work in progress with Alison Cooke of Aberystwyth University on end-user perceptions of information quality in the networked environment).  However, some of the existing results may help indirectly.  For example, a number of the surveys found that users were highly satisfied with the quality of the services as they stood (e.g. OMNI, EEVL, SOSIG), suggesting that users were satisfied with the selection criteria being used by those services.  Similarly, some surveys yielded information on the nature of the user population, which could imply that certain types and qualities of resources would be required to meet certain needs.  The EEVL survey produced a list of the types of information that users were searching for on the Internet providing a focus for resource selection.

Appendix VIII: E-Mail questionnaire sent to subject gateways, July 1996.

[Customised introduction to each mail]

1. Are documents/services actively selected for inclusion in your
service/gateway?

Yes/No

If *No* you will probably not need to fill in any more details.
If *Yes* would it be possible to get some answers to the following? Feel
free to expand if necessary.

2. Who does the selection?

3. What criteria are used and are they published?

4. What issues do you think have been raised by the selection process?

5. Could resource selection be improved in any way? Have any refinements
been made to the selection process?

6. How do you ensure the up-to-dateness of the information resources you
include with relation to:
   a. new resources?

   b. currently included resources?  

7. Are there any constraints on the selection process?

8. How can selection be applied to resource descriptions created by
gathering software?

9. Have you found any publications/WWW sites particularly useful in
formulating resource selection guidelines?

If these questions are not relevant, please include any additional
comments below.

Thank-you very much for your help with this.

Michael Day
Research Officer
UKOLN
University of Bath
Bath BA2 7AY, UK.

E-mail: lismd@bath.ac.uk
Selection criteria issues raised by the subject gateways
Replies were received from: ADAM, BUBL, EELS, EEVL,  NBW (KB), OMNI, RUDI and SOSIG.

The following issues were identified:
Granularity
Granularity is central to the problem of quality control.  The basic problem is knowing at what level to catalogue the resource.  Briefly, resources could be catalogued at a high level (say a service, or collection of documents) or a low level (an individual document or data-set).  The higher level would be more economical on time and effort than lower level cataloguing, but is less good for those searching the database.  This is not strictly speaking quality criteria, but judgements in this area will have an impact on the selection criteria used. 

One of the responses stated: 
“Granularity - The problem of defining what a resource is.  This is a much wider issue than the quality one, and goes to the heart of the problem of indexing things on the Internet.  Resources are often inter-linked so there is a problem with knowing where to aim the selection process.  The key question is: at what level should selection take place?  Individual Files? The Server?  This may not seem to be particularly relevant to quality selection issues, but quality guidelines may be different for different levels”.

Another respondent noted that there was a “lack of in-depth work to formalise criteria according to different types of resources (e.g. electronic journals versus image databases, etc.)”.
Overlap between subjects within and between subject gateways
As there is likely to be some overlap between different fields in a particular subject gateway, the same resource could be unintentionally included more than once in the same database, assuming that different people are doing the indexing.  This need not be a problem if all the “cataloguers” check the database before adding a new resource.  For similar reasons, there will be some duplication of effort in subject areas which straddle more than one of the subject gateways, e.g. architecture in EELS/ADAM/RUDI; sociology of health care in SOSIG/OMNI.
Exclusion of sites
One of the eLib subject gateways questioned exclusion criteria.  They mentioned that resources excluded by the selection process could be of some interest to users of the database.  Specifically, resources which consist almost entirely of links to other resources may have some use in the user community.  Similarly, badly organised sites may be better than nothing, e.g. poor image libraries might be better than none in the context of an arts subject gateway.  If subject gateways only select good quality resources, to what extent do they take away the user’s ability to decide for themselves whether to consult a resource?
Increased co-operation with information providers
Several subject gateways noted the need for increased co-operation with the information providers themselves.  Some services already contact information providers to fill-in gaps in their knowledge about a resource (if for example a resource was missing a date) and would be prepared to contact them again to check whether resources were going to be updated regularly.  Others would like information providers to contact them with details of new or updated sites when necessary or to make some comments on the resource description made by the service. 
End-user perceptions of information quality in an network environment
One respondent noted that criteria currently used were not validated with reference to their suitability for the network environment or to the subject areas being covered by the database itself.  One of the subject gateways (OMNI) is conducting research into end user perceptions of information quality.
Use of gatherer software/metadata
It was generally agreed that robotic gatherers could not be used as the sole arbiter for selection.  Selection is a complex action needing some human intervention.  However, it might be possible for a gatherer to make some ‘pre-selection’ to which the subject specialist cataloguers could approve and add description.  Alternatively, gatherers could be used to retrieve document level resources from sites already acknowledged to be of high quality.  If embedded metadata were included in the original resources being described, a metadata aware gatherer could take this information and place it in the relevant parts of the resource description for later approval by the subject gateways.  This might speed up the cataloguing of resources.  One service suggested some type of automated ‘current awareness’ service, using automated link and content checkers, to identify new or altered pages on selected relevant sites.
Ensuring that a resource is up-to date
Checking that a resource is up-to-date is problematic.  Most of the services currently use an automated link checker, but there is no consistent way of ensuring that a site is kept up-to-date.  With a large database, it is not possible to check resources individually.  Here there is need for either increased co-operation between subject gateways and information providers or use of a review-by-date in resource descriptions - as currently used by SOSIG for conference information.
Selection expertise and subjectivity
A couple of the subject gateways commented that knowing exactly what was relevant and what constituted “quality” in a particular subject area was difficult - even when subject-specialist librarians were doing the selection.  Another was concerned by the amount of importance given to the provenance of information in the selection process.  It was noted that resource selection was largely subjective.  One respondent noted an “initial lack of confidence on behalf of librarians / information professionals to undertake evaluation activities in the network environment”.  The same person also noted that little work has been done on “developing procedures and methodologies to enable subject specific contribution to the process”.
Constraints
The main constraints mentioned were the amount of personnel and time allocated to the selection process.  It was noted by one respondent that the learning curve associated with the selection process was in itself time consuming.  Other perceived constraints are the large amount of potentially useful information available on the Internet and the lack of background information supplied with some resources.
Copyright
One of the services mentioned the problem of copyright.  Although this is usually a problem for persons who want to add copyright information to their own services, it is worth noting that many current lists of selection criteria do not usually stipulate that a resource included in a subject gateway should not break international copyright law.
National resources
Some of the subject gateways have a particular interest in resources from geographical areas.  For example EELS concentrates on Nordic resources and the eLib services on the UK and Europe.  Different selection criteria might need to be devised for resources perceived to be of particular utility for the international user community.

Appendix IX: Personnel involved in testing 
The following personnel from each organisation were involved in the testing:
 
Biz/ed
Catherine Sladen, Research Officer for Biz/ed, based at Bristol University, UK.

EELS
Kjell Jansson, EELs editor

with additional comments from Traugott Koch, Electronic Information Services Librarian at the University of Lund, Sweden.

KB
Marianne Peereboom, Nederlandse Basisclassificatie Web, National Library of the Netherlands.

quality criteria only:

Gerard Baltussen, Head of Cataloguing and Acquisitions at the University Library in Utrecht, Netherlands

Dr. M.L.B. Blom, a librarian in the Faculty of Arts at Utrecht University, Netherlands

Jeroen Vervliet, Reference Librarian & Staff Member, Leiden University Library, Netherlands

SOSIG
Debra Hiom, SOSIG Research Officer, Bristol University, UK

Emma Worsfold, SOSIG Research Officer, Bristol University, UK

Appendix X: Rich pictures and models
Three early iterations of the model are available as:

 <URL:http://sosig.ac.uk/desire/mdlv1_1.gif>
<URL:http://sosig.ac.uk/desire/mdlv1_2.gif>
<URL:http://sosig.ac.uk/desire/mdlv1_3.gif>

The pre test model is available as:

 <URL:http://sosig.ac.uk/desire/mdlv1_4.gif>

The post-test model is available as:

 <URL:http://sosig.ac.uk/desire/mdlv1_5.gif>

Part IV: Bibliography and references

Some of these references are taken from the Bibliography on Evaluating Internet Resources compiled by Nicole Auer. <URL: http://refserver.lib.vt.edu/libinst/critTHINK.HTM>[Accessed: 16 Dec 1996]

Anand, P., Chong Ju Choi, and Grint, K., 1993, Knowledge, quality surrogates and idea adoption. Management Research Papers MRP 93/3. Oxford: Templeton College, the Oxford Centre for Management Studies.

Argus Associates, 1996, Clearinghouse: information: rating system. 
<URL: http://www.clearinghouse.net/docs/ratings.html> [Accessed: 28 Nov. 1996]

Armstrong, C.J., 1995a, Database information quality. Library & Information Briefings, no. 62. London: South Bank University.

Armstrong, C.J., 1995b, Database labelling: a CIQM meeting, 4th May 1995, the Library Association. 
<URL: gopher://ukoln.bath.ac.uk:7070/00/BUBL_Main_Menu/H/H2/H2C/H2C28/H2C28010_-_Database_Labelling_Meeting_Report> [Accessed: 28 Nov. 1996]

Armstrong, C.J., 1995c, Do users really want database labelling? Assignation: Aslib Social Sciences Information Group Journal, Vol. 13, no. 1, pp. 15-17.

Armstrong, C.J., 1995d, Database quality criteria. 
<URL: gopher://ukoln.bath.ac.uk:7070/00/BUBL_Main_Menu/H/H2/H2C/H2C28/H2C28002_-_Database_Quality_Criteria > [Accessed: 28 Nov. 1996] 
<URL: http://mystic.biomed.mcgill.ca/MedinfHome/REPOSIT/dbquality.txt> [Accessed: 28 Nov. 1996]

Armstrong, C.J., 1996, The quality of publicly-available databases: WYSIWYG or what? [An updated version of a paper first given at the BULISC 95 Conference]. In: Routes to quality. (Bournemouth University Occasional Papers on Library and Information Services, 1). Bournemouth: Bournemouth University. 
<URL: gopher://ukoln.bath.ac.uk:7070/00/BUBL_Main_Menu/H/H2/H2C/H2C28/96%23> [Accessed: 28 Nov. 1996]

Art, Design, Architecture & Media Information Gateway (ADAM), 1996, About ADAM. 
<URL: http://adam.ac.uk/adam/> [Accessed: 29 Nov. 1996]

Ashton, T.C., 1994, National and International Standards. In: Lock, D., (ed.), Gower handbook of quality management, 2nd ed. Aldershot: Gower, pp. 133-156.

Auer, N., 1996, Bibliography on evaluating Internet resources.
<URL: http://refserver.lib.vt.edu/libinst/critTHINK.HTM> [Accessed: 4 Dec. 1996]

Bartelstein, A. and Zald, A., 1996, R545: Teaching students to think critically about Internet resources. University of Washington C&C/UWired Computer Training.
<URL: http://weber.u.washington.edu/~libr560/NETEVAL/index.html> [Accessed: 29 Nov. 1996]

Basch, R., (ed.), 1995, Electronic information delivery: ensuring quality and value. Aldershot: Gower.

Basch, R., 1990a, Databank software for the 1990s and beyond - Part 1: the users’ wish list.  Online, Vol.14, no. 2, pp. 17-24.

Basch, R., 1990b, Databank software for the 1990s and beyond - Part 2: the online services respond.  Online, Vol.14, no. 3, pp. 15-21.

Basch, R., 1990c, Measuring the quality of the data: report on the Fourth Annual SCOUG Retreat. Database Searcher, Vol. 6, no. 8, pp. 18-23.

Basch, R., 1992, An overview of quality assurance issues. In: Armstrong, C.J. and Hartley, R.J., (eds.), Database 2000: UKOLUG state-of-the-art conference, 1992. Oxford: Learned Information, pp. 85-91.

Bendell, T, Boulter, L. and Kelly, J., 1993, Benchmarking for competitive advantage. London: Pitman (Financial Times).

Bergman, B. and Klefsjö, B., 1994, Quality: from customer needs to customer satisfaction. London: McGraw-Hill; Lund: Studentlitteratur.

Berners-Lee, T., 1992-1995, Style guide for online hypertext. 
<URL: http://www.w3.org/pub/WWW/Provider/Style/Overview.html> [Accessed: 28 Nov. 1996]

Bodi, S., 1995, Scholarship or propaganda: how can librarians help undergraduates tell the difference? Journal of Academic Librarianship, Vol. 21, no. 1, January, pp. 21-25.

Boekhorst, P. te, 1995, Measuring quality: the IFLA guidelines for performance measurement in
academic libraries. IFLA Journal, Vol. 21, no. 4, pp. 278-281.

Bradshaw, R., 1996a, Criteria for including resources [in ADAM]. E-mail [from Tony Gill] to list: ADAM-NEWS <adam-news@mailbase.ac.uk> , 4 Apr. 
<URL: http://www.mailbase.ac.uk/lists-a-e/adam-news/1996-04/0000.html> [Accessed: 28 Nov. 1996]

Bradshaw, R., 1996b, Guidelines for the selection of resources for inclusion in the ADAM database. 
<URL: http://adam.ac.uk/adam/reports/select/> [Accessed: 29 Nov. 1996]

Brockman, J.R., 1992, Just another management fad? The implications of TQM for library and information services. Aslib Proceedings, Vol. 44, no. 7/8, July/August, pp. 283-288.

Brophy, P., Coulling, K. and Melling, M., 1993, Quality management: a university approach. Aslib Information, Vol. 21, no. 6, June, pp. 246-248.

BS 4778: Part 1: 1987. Quality vocabulary. London: British Standards Institution.

BS 5750: Part 0: Section 0.1: 1987. Quality systems - Part 0: principal concepts and applications - Section 0.1: guide to selection and use. London: British Standards Institution. [British Standards Service: Microfiche 07357-F11. Bracknell: Technical Indexes].

BS 5750: Part 8: 1991. Quality systems - Part 8: guide to quality management and quality systems elements for services. London: British Standards Institution. [British Standards Service: Microfiche 07608-A3. Bracknell: Technical Indexes].

BS EN ISO 8404: 1995. Quality management and quality assurance - vocabulary. London: British Standards Institution. [British Standards Service: Microfiche 07691-A3. Bracknell: Technical Indexes].

BS EN ISO 9000-1: 1994. Quality management and quality assurance standards - Part 1: guidelines for selection and use. London: British Standards Institution. [British Standards Service: Microfiche 07651-D8. Bracknell: Technical Indexes].

Bullivant, J.R.N., 1994, Benchmarking for continuous improvement in the public sector. Harlow: Longman Information & Reference.

Camp, R.C, 1989, Benchmarking: the search for industry’s best practices that lead to superior performance. Milwaukee, Wis.: ASQC Quality Press.

Carbone, P., 1995, The committee draft of International Standard ISO CD 11620 on library
performance indicators. IFLA Journal, Vol. 21, no. 4, pp. 274-277.

Caruana, A. and Pitt, L., 1994, Excellence, market orientation some aspects of service quality and their effect on performance in service companies: propositions and a model. Working Paper Series HWP 9406. Henley on Thames: Henley Management College

Caruana, A., Morris, M. and Pitt, L., 1994, Excellence, market orientation and service quality: are they related? Working Paper Series HWP 9419. Henley on Thames: Henley Management College.

Cassel, R., 1995, Selection criteria for Internet resources. College and Research Libraries News, Vol. 56, no. 2, February, pp. 92-93.

Caywood, C., 1995, Library selection criteria for WWW resources. 
<URL: http://duckdock.acic.com/carolyn/criteria.htm> [Accessed:. 28 Nov. 1996]

Centre for Information Quality Management, 1996, A description of database labels. 
<URL: http://www.fdgroup.co.uk/ciqmlbl2.htm> [Accessed: 28 Nov. 1996]

Chapman, A., 1994, Up to standard? A study of the quality of records in a shared cataloguing database. Journal of Librarianship and Information Science, Vol. 26, no. 4, December, pp. 201-210.

Checkland, P.B., 1981, Systems thinking, systems practice. Chichester: Wiley.

Checkland, P.B. and Scholes, J., 1990, Soft Systems Methodology in action. Chichester: Wiley.

Ciolek, T.M., 1995, Ensuring high quality in multifaceted information services. Paper presented to AUUG’95 and Asia-Pacific WWW’95 Conference, Sydney, Australia, 18-21 September 1995. 
<URL: http://coombs.anu.edu.au/SpecialProj/QLTY/EnsuringQlty.html> [Accessed: 29. Nov. 1996]

Ciolek, T.M., 1996a, Today’s WWW - tomorrow’s MMM? The specter of multi-media mediocrity. IEEE Computer, Vol. 29, no. 1, January, pp. 106-108. 
<URL: http://www.computer.org/pubs/computer/kiosk/OI/kiosk.htm> [Accessed: 26 June 1996]

Ciolek, T.M., 1996b, The six quests for the electronic grail: current approaches to information quality in WWW resources. Revue Informatique et Statistique dans les Sciences Humaines (RISSH) (Centre Informatique de Philosophie et Lettres, Université de Liège), No. 1-4, 1996. 
<URL: http://coombs.anu.edu.au/SpecialProj/QLTY/TMC/QuestMain.html> [Accessed: 29 Nov. 1996]

Ciolek, T.M., (ed.), 1994, Quality WWW resources.
<URL: http://coombs.anu.edu.au/SpecialProj/QLTY/QltyWWW.html> [Accessed: 29 Nov. 1996]

Ciolek, T.M., (ed.), 1995, Criteria used to select links for resources’ catalogues. 
<URL: http://coombs.anu.edu.au/SpecialProj/QLTY/QltyLinks.html> [Accessed: 29 Nov. 1996].

Ciolek, T.M., (ed.), 1996a, Quality, guidelines & standards for Internet resources. 
<URL: http://coombs.anu.edu.au/SpecialProj/QLTY/QltyHome.html>  [Accessed: 29 Nov. 1996]

Ciolek, T.M., (ed.), 1996b, Information Quality WWW Virtual Library: the Internet guide to construction of quality online resources. 
<URL: http://coombs.anu.edu.au/WWWVL-InfoQuality.html> Accessed: 29 Nov. 1996]

Ciolek, T.M., (ed.), 1996c, Information quality: some definitions.
<URL: http://coombs.anu.edu.au/SpecialProj/QLTY/QltyDefinitions.html> [Accessed: 29 Nov. 1996]

Clark, F.A., 1992, Quality and service: a key focus for performance in the public sector. Working Paper Series HWP 11/92. Henley on Thames: Henley Management College.

Collins, B.R., 1996a, WebWatch column. Library Journal, Vol. 121, no. 2, February 1, p. 32

Collins, B.R., 1996b, Beyond cruising: reviewing. Library Journal, Vol. 121, no. 3, February 15, pp. 122-124.

Cooke, A., et al., undated, Developing a technique for assessing the quality of networked information. Unpublished paper. Aberystwyth: University of Wales Aberystwyth, Department of Information and Library Studies.

Courtois, M.P., 1996, Cool tools for Web searching: an update. Online, Vol. 20, no. 3, May/June, pp. 29-36.

Courtois, M., Baer, W. and Stark M., 1995, Cool tools for searching the Web: a performance evaluation. Online, Vol. 19, no.6, November/December, pp. 14-32.

Cox, J., 1995, Organising Medical Networked Information: profile of the OMNI Project. Paper delivered at: 1st Annual OMNI Seminar, Wellcome Centre for Medical Science, London, 30 November 1995. 
<URL: http://omni.ac.uk/general-info/launch/serv-jc.html> [Accessed: 29 Nov. 1996]

Cox, M.D, 1996, LogLink Logistics: web page review criteria. 
<URL: http://www.commerce2000.com/logistics/criteria.html-ssi> [Accessed 28 Nov. 1996]

Crosby, P.B., 1979, Quality is free: the art of making quality certain. New York: McGraw-Hill.

December, J, 1994, Challenges for Web information providers. Computer-Mediated Communication Magazine, Vol. 1, no. 6, pp. 8-14. 
<URL: http://sunsite.unc.edu/cmc/mag/1994/oct/webip.html> [Accessed: 28 Nov. 1996]

December, J., 1996, Web development quality.
<URL: http://www.december.com/web/develop/quality.html> [Accessed: 29. Nov. 1996]

December, J. and Randall, N., 1996, The World Wide Web unleashed. 3rd ed. Indianapolis, Ind.: Sams.Net Publishing.

Dillon, M., Jul, E., Burge, M. and Hickey, C., 1993, Assessing information on the Internet: toward providing library services for computer mediated communication. Dublin, Ohio: OCLC Office of Research and Special Projects.
<URL: http://www.oclc.org:5046/oclc/research/publications/aii/table.html> [Accessed: 4 Dec 1996]

Doran, K., 1995, The Internot: helping library patrons understand what the Internet is not (yet). Computers in Libraries, Vol. 15, no. 6, June, pp. 22-26.

Ebbinghouse, C., 1995a, SCOUG 1995 Retreat - Quality on the Internet: daily report … Saturday. 
<URL: http://www.gslis.ucla.edu/SCOUG/saturday.html> [Accessed: 19 Sep. 1996]

Ebbinghouse, C., 1995b, SCOUG 1995 Retreat - Quality on the Internet: daily report … Sunday. 
<URL: http://www.gslis.ucla.edu/SCOUG/sunday.html> [Accessed: 19 Sep. 1996]

Edinburgh Engineering Virtual Library (EEVL), 1995, EEVL (Edinburgh Engineering Virtual Library): a  proposal to ELP/ISSC as part of the Electronic Libraries Programme (Area 6: Access to Network Resources). 20 June.

Edinburgh Engineering Virtual Library (EEVL), 1996, Quality and resource selection, v 1.0. (EEVL team manual, 3.1) 18 January. 

Elementary School Student’s Internet Gateway, 1995, Evaluating information: some questions to help you judge online information. 
<URL: http://volvo.gslis.utexas.edu:80/~clig/evalinfo.html> [Accessed: 28 Nov. 1996]

EN 29000: 1987. Quality management and quality assurance standards. Brussels: European Committee for Standardization (CEN).

EN 29004-2: 1993. Quality  management and quality systems elements - Part 2: guidelines for services. Brussels: European Committee for Standardization (CEN).

EN ISO 8404: 1995. Quality management and quality assurance - vocabulary. Brussels: European Committee for Standardization (CEN).

EN ISO 9000-1: 1994. Quality management and quality assurance standards - guidelines for selection and use. Brussels: European Committee for Standardization (CEN).

Farber, M., 1994, The quality of information on Internet. Computer Networks and ISDN Systems, Vol. 26, Suppl. 2, pp. S75-S78.

Fisher, T., 1995, Evaluation of an Internet information resource: [Emergency Medical Services]. For: Diploma in Library Science, Department of Library and Information Studies, Victoria University of Wellington, N.Z.. 
<URL: http://www.vuw.ac.nz/~thelma14/eval.html> [Accessed:29 Nov. 1996]

Ford, A., 1995, Spinning the Web: how to provide information on the Internet. London: International Thomson.

Gale Research, 1996, Rating criteria [for Cyberhound]. 
<URL: http://www.thomson.com/cyberhound/ratings.html> [Accessed: 28 Nov. 1996]

Garman, N., 1995, When online is not enough. Online, Vol. 19, no. 3, May/June, pp. 6-7.

Gorman, M., 1995, The corruption of cataloging. Library Journal, Vol. 120, no. 15, September 15, pp. 32-34.

Grassian, E., 1996, Thinking critically about World Wide Web resources. 
<URL: http://www.ucla.edu/campus/computing/bruinonline/trainers/critical.html> [Accessed: 28 Nov. 1996]

Grönroos, C., 1984a, Strategic management and marketing in the service sector. Lund: Studentlitteratur; Bromley: Chartwell-Bratt.

Grönroos, C., 1984b, A service quality model and its marketing implications. European Journal of Marketing, Vol. 18, no. 4, pp. 36-44.

Gurn, R.M., 1995, Measuring information providers on the Internet. Computers in Libraries, Vol. 15, no. 1, January, p. 42.

Harter, S.P., 1986,  Online information retrieval: concepts, principles and techniques. London: Academic Press.

Harvard, F., 1994, BS ISO 9000 Certification. In: Lock, D., (ed.), Gower handbook of quality management, 2nd ed. Aldershot: Gower, pp. 219-228.

Hernon, P., 1995, Disinformation and misinformation through the Internet: findings of an exploratory
study. Government Information Quarterly, Vol. 12 no. 2, pp. 133-139.

Info Filter Project, 1996, Review title [i.e. Review template (model HTML format)]. 
<URL: http://www.usc.edu/users/help/flick/Infofilter/template.html> [Accessed: 28 Nov. 1996]

Information Market Observatory (IMO), 1995, The quality of electronic information products and services. IMO Working Paper 95/4. Luxembourg: IMO, September. 
<URL: http://www2.echo.lu/impact/imo/9504.html> [Accessed: 29 Nov. 1996]

ISO 8402: 1986. Quality vocabulary. Geneva: International Organisation for Standardization.

ISO 8402: 1994. Quality management and quality assurance - vocabulary.  Geneva: International Organisation for Standardization.

ISO 8402: 1994. Quality management and quality assurance - vocabulary. Geneva: International Organisation for Standardization. 

ISO 9000: 1987. Quality management and quality assurance standards - guidelines for selection and use. Geneva: International Organisation for Standardization. 

ISO 9004-2: 1991. Quality management and quality systems elements - Part 2: guidelines for services. Geneva: International Organisation for Standardization.

Jacsó, P., 1992, CD-ROM software, dataware and hardware: evaluation, selection and illustration. (Databases searching series, no. 4). Englewood, Colo.: Libraries Unlimited.

Janicke Hinchliffe, L., 1996, Resource selection and information evaluation. 
<URL: http://alexia.lis.uiuc.edu/~janicke/Evaluate.html> [Accessed: 28 Nov. 1996]

Jansson, K., 1996, About EELS. 
<URL: http://www.ub2.lu.se/eel/about.html> [Accessed: 28 Nov. 1996]

Johannsen, C.G., 1995, Quality management and innovation:  findings of a Nordic quality management
survey. Libri, Vol. 45, No. 3/4, September/December, pp. 131-144.

Johnson, P., 1996, Selecting electronic resources: developing a local decision making matrix. Cataloguing & Classification Quarterly, Vol. 22, no. 3/4. Pp. 9-24.

Joint Funding Councils’ Libraries Review Group, 1993, Report of the Libraries Review Group. Bristol: Higher Education Funding Council for England. 
<URL: http://ukoln.bath.ac.uk/follett/follett_report.html> [Accessed: 28 Nov. 1996]

Joint Information Systems Committee (JISC), 1994, Follett Implementation Group on Information Technology: framework for progressing the initiative. JISC Circular 4/94 - 3 August. Bristol: JISC. 
<URL: http://ukoln.bath.ac.uk/figit/figit-4-94.html> [Accessed: 28 Nov. 1996]

Jones, D., 1996, Critical thinking in an online world.
<URL: http://www.library.ucsb.edu/untangle/jones.html> [Accessed: 4. Dec 1996]

Juran, J.M., 1988, The quality function. In: Juran, J.M. and Gryna, F.M., (eds.), Juran’s quality control handbook, 4th ed. New York: McGraw-Hill, pp. 2.1-2.13.

Kinnell, M., 1995, Quality management and library and information services: competitive
advantage for the information revolution. IFLA Journal, Vol. 21, no. 4,  pp. 265-273.

Kirk, E.E., Evaluating information found on the Internet.
<URL: http://milton.mse.jhu.edu:8001/research/education/net.html> [Accessed: 4 Dec. 1996]

Knight, J.P. and Hamilton, M.T., 1996, Overview of the ROADS software. LUT CS-TR 1010. Loughborough: Loughborough University of Technology, Department of Computer Studies, March. 
<URL: http://www.roads.lut.ac.uk/Reports/arch/arch.html> [Accessed: 28 Nov. 1996]

Koch, T., 1996, Spotlight on BUBL. Ariadne, No. 1, January. 
<URL: http://www.ukoln.ac.uk/ariadne/issue1/bubl/> [Accessed: 28 Nov. 1996]

Koopman, A. and Hay, S., 1994, Swim at your own risk - no librarian on duty: large-scale application of Mosaic in an academic library. Conference paper, September 15, 1994: Electronic Proceedings of the Second World Wide Web Conference ‘94: Mosaic and the Web.
<URL: http://www.ncsa.uiuc.edu/SDG/IT94/Proceedings/LibApps/hay/WWWPap.html> [Accessed: 28 Nov. 1996]

Kovacs, D., Schloman, B.F. and McDaniel J.A., 1994, A model for planning and providing reference services using Internet resources. Library Trends, Vol. 42, no. 4, Spring, pp. 638-647.

Lemay, L, 1995, Teach yourself Web publishing with HTML in a week. Indianapolis, Ind.: Sams.Net Publishing.

Lewis, P.J., 1992, Rich picture building in the soft systems methodology. European Journal of Information Systems, Vol. 1, no. 5, pp. 351-360.

Liu Jian, 1996, Understanding WWW search tools.  
<URL: http://www.indiana.edu/~librcsd/search/> [Accessed: 28 Nov. 1996]

Lynch, P.J., Web style manual.
<URL: http://info.med.yale.edu/caim/StyleManual_Top.HTML> [Accessed: 28 Nov. 1996]

McClure, C.R., 1994, User-based data collection techniques and strategies for evaluating networked information services.  Library Trends, Vol. 42, no. 4, Spring, pp. 591-607.

McLachlan, K., 1996, WWW CyberGuide ratings for content evaluation. 
<URL: http://www.cyberbee.com/guide1.html>. [Accessed: 28 Nov. 1996]

Makulowich, J., 1996, Quality control on the Net. Database, Vol. 19, no. 1, February/March, pp. 93-94.

Martin, D., 1993, Total Quality Management. Library & Information Briefings, no. 45. London: South Bank University.

Mathews, B.P., 1995, Back to basics in service quality. TBS Monograph Series, GEN 19. Normanby: University of Teesside.

Medawar, K., 1995, Database quality: a literature review of the past and a plan for the future. Program, Vol. 29, no. 3, July, pp. 257-272.

Memorial Library, Department of Education, 1996, Internet Source Validation Project.
<URL: http://www.stemnet.nf.ca/Curriculum/Validate/> [Accessed: 4 Dec. 1996]

Moffat, M., 1996a, An EEVL solution to engineering information on the Internet. [Paper presented to: Aslib Electronics Group 38th Annual Conference 15-17 May]. Aslib Proceedings, Vol. 48, no. 6, June, pp. 147-150.  
<URL: http://www.eevl.ac.uk/pub3.html> [Accessed: 28 Nov. 1996]

Moffat, M., 1996b, Re: Standards and quality criteria. E-mail to list: COUSNS <cousns@mailbase.ac.uk> , 3 June.  
<URL: http://www.mailbase.ac.uk/lists-a-e/cousns/1996-06/0000.html> [Accessed: 28 Nov. 1996]

Morrison, S.J., 1994, Managing quality: an historical review. In: Dale, B.G., (ed.), Managing quality, 2nd ed. Hemel Hempstead: Prentice-Hall, pp. 41-79.

National Center for Supercomputing Applications (NCSA), 1996, A beginner’s guide to HTML. 
<URL: http://www.ncsa.uiuc.edu/General/Internet/WWW/HTMLPrimer.html> [Accessed: 28 Nov. 1996]

Nederlands Basisclassificatie Web (NBW), 1996, Richtlijnen voor de selectie van Bronnen. 27 March 1996.  
<URL: http://www.konbib.nl/nbw/werk/selectie.htm> [Accessed: 28 Nov. 1996]

Nicolson, D., 1993, BUBL and the development of the UK LIS networking community. VINE, No. 93, December,  pp.12-17.

Nielsen, J., 1995-96, The Alertbox: current issues in user interface design.
<URL: http://www.sun.com/columns/alertbox.html> [Accessed: 29 Nov. 1996]

Norton, R. and Ellis, D., 1993, Implementing BS 5750. Aslib Information, Vol. 21, no. 6, June, pp. 242-245

O’Reilly & Associates and Trish Information Systems, 1995, Defining the Internet opportunity.
<URL: http://www.ora.com/www/info/research/users/index.html> [Accessed: 28 Nov. 1996]

Oberman, C., 1991, Avoiding the cereal syndrome, or critical thinking in the electronic environment. Library Trends, Vol. 39, no. 3, Winter, pp. 189-202.

Oberman, C., 1995, Unmasking technology: a prelude to teaching. Research Strategies, Vol. 13, no. 1, Winter, pp. 34-39.

Organising Medical Networked Information (OMNI) Consortium, 1994, OMNI: Organising Medical Networked Information: a proposal to ISSC as part of the Follett Implementation Group on Information Technology programme (Area 6: Access to Network Resources) submitted in response to JISC circular 4/94.  
<URL: http://omni.ac.uk/general-info/proposal.html> [Accessed: 28 Nov. 1996]

Organising Medical Networked Information (OMNI) Consortium, 1996a, OMNI Newsletter, no. 1, 12 April.  
<URL: http://omni.ac.uk/general-info/newsletter/nl1.html> [Accessed: 28 Nov. 1996]

Organising Medical Networked Information (OMNI) Consortium, 1996b, OMNI: Organising Medical Networked Information: evaluating resources for OMNI.  
<URL: http://omni.ac.uk/general-info/reselec.html> [Accessed: 28 Nov. 1996]

Ormondroyd, J., Engle, M. and Cosgrave, T., 1996. How to critically analyze information sources. Cornell University Library. 
<URL: http://www.library.cornell.edu/okuref/research/skill26.htm> [Accessed: 28 Nov. 1996]

Page, M. and Kesselman, M., 1994, Teaching the Internet: challenges and opportunities. Research Strategies, Vol. 12, no. 3, Summer, pp. 157-167

Pagell, R.A., 1995, Quality and the Internet: an open letter. Online, Vol. 19, no. 4, July/August, pp. 7-9.

Parasuraman, A., Zeithaml, V.A. and Berry, L.L., 1985, A conceptual model of service quality and its implications for future research. Journal of Marketing, Vol. 49, pp. 41-50.

Parasuraman, A., Zeithaml, V.A. and Berry, L.L., 1988, SERVQUAL: a multiple-item scale for measuring customer perceptions of service quality. Journal of Retailing, Vol. 64, pp. 12-40.

Parasuraman, A., Zeithaml, V.A. and Berry, L.L., 1991, Refinement and reassessment of the SERVQUAL scale. Journal of Retailing, Vol. 67, pp. 420-450.

Peters, T. and Waterman, R.H., 1982, In search of excellence. New York: Harper & Row.

Pilling, S., 1996, Putting the customer first: total quality and customer service at the British Library Document Supply Centre. Interlending and Document Supply, Vol. 24, no. 2, pp. 11-16.

Pitt, L., Caruana, A. and Money, A., 1994, Managing customer expectations of service quality: does it make a difference? Working Paper Series HWP 9404. Henley on Thames: Henley Management College

Pratt, G.F., Flannery, P. and Perkins, C.L.D., 1996, Guidelines for Internet resource selection. College and Research Libraries News, Vol. 57, no. 3, March, pp. 134-135.

Rader, H., Reinhart, B. and Thompson, G., 1990, Evaluating information: a basic checklist. (Item number A127). Chicago, Ill.: ALA.

Rettig, J., 1995,  Putting the squeeze on the information firehose: the need for ‘Neteditors and ‘Netreviewers.  
<URL: http://www.swem.wm.edu/firehose.html> [Accessed: 4 Dec 1996]

Rettig, J., 1996, Beyond cool: analog models for reviewing digital resources. Online, Vol. 20, no. 5, September/October, pp. 52-64.

Rosenfeld, L.B., 1994a, Guides, clearinghouses, and value-added repackaging: some thoughts on how librarians can improve the Internet. Reference Services Review,  Vol. 22, no. 4, Winter, pp. 11-16.

Rosenfeld, L.B., 1994b, Resource evaluation: assessing resource quality for ‘Building Subject Guides to Internet Resources’.  
<URL: http://http2.sils.umich.edu/~lou/bsgeval.html> [Accessed: 29 Nov. 1996]

Rothery, B, 1993, ISO 9000, 2nd ed. Aldershot: Gower.

Rowley, J., 1996, Implementing TQM for library services: the issues. Aslib Proceedings, Vol. 48, no. 1, pp. 17-21.

Santa Vicca, E.F., 1994, The Internet as a reference and research tool: a model for educators. Reference Librarian, No. 41-42, pp. 225-236.

Schlesinger, L.A. and Heskett, J.L., 1991, The service-driven service company. Harvard Business Review, Vol. 69, no. 5, September/October, pp. 71-81.

Scholz, A., 1996, Evaluating World Wide Web information.
<URL: http://thorplus.lib.purdue.edu/research/classes/gs175/3gs175/evaluation.html> [Accessed: 4 Dec. 1996]

Seay, T., Seaman, S. and Cohen, D., 1996, Measuring and improving the quality of public services: a hybrid approach. Library Trends, Vol. 44, no. 3, Winter, pp. 464-490.

Smith, A., 1996a, Criteria for evaluation of Internet information resources.
<URL: http://www.vuw.ac.nz/~agsmith/evaln/> [Accessed: 29 Nov. 1996]

Smith, A., 1996b, Selection criteria for Internet information resources: a poll of members of info-quality-l.
<URL: http://www.vuw.ac.nz/~agsmith/evaln/poll.htm> [Accessed: 29 Nov. 1996]

Social Science Information Gateway (SOSIG), 1996, Overview of the Service. July. 
<URL: http://www.sosig.ac.uk/Docs/sos.html> [Accessed: 29 Nov. 1996]

SOFWeb, 1996, Research and the Internet.
<URL: http://www.dse.vic.gov.au/netiget2.htm> [Accessed: 30 April 1996]

Southern California Online User Group (SCOUG), 1995, Quality scales revisited:  SCOUG tames the Internet. Report from the Southern California Online Users Group 9th Annual Retreat, July 28 - 30 1995, Santa Barbara, California. 
<URL: http://www.gslis.ucla.edu/SCOUG/retreat95.html> [Accessed: 19 Sep. 1996]

Starr, S.S., 1994, Evaluating physical science references on the Internet. Reference Librarian, No. 41-42, pp. 261-273.

Stepno, R. and Henshaw, R., 1995, Quality of information … and disinformation online. 
<URL: http://blake.oit.unc.edu/~rbstepno/disinfo.html> [Accessed: 29 Nov. 1996]

Stoker, D. and Cooke, A., 1993, Evaluation of networked information sources. In: Helal, A.H. and Weiss, J.H. (eds.), Opportunity 2000: understanding and serving users in an electronic library: 15th International Essen Symposium … Festschrift in honour of Herbert S. White. Publications of Essen University Library, 15. Essen: Universitätsbibliothek Essen, pp. 287-312.

Sun Microsystems, 1995, Guide to Web style: quality. 
<URL: http://www.sun.com/styleguide/tables/Quality.html> [Accessed: 29 Nov.1996]

Thomas, S.E., 1996, Quality in bibliographic control. Library Trends, Vol. 44, no. 3, Winter, pp. 491-505.

Tillman, H.N., 1996, Evaluating quality on the Net. 
<URL: http://www.tiac.net/users/hope/findqual.html> [Accessed: 29 Nov. 1996].

US Department of Education, 1996, World Wide Web (WWW) server standards and guidelines.
<URL: http://inet.ed.gov/~kstubbs/wwwstds/index.html> [Accessed: 4 Aug. 1996]

Victoria University of Wellington, Department of Library and Information Studies, 1995b, Criteria for evaluation of Internet information resources. 
<URL: http://www.vuw.ac.nz/dlis/courses/847/m2resevl.html> [Accessed: 29 Nov. 1996]

Victoria University of Wellington, Department of Library and Information Studies, 1995a, Module 2: Evaluating and locating Internet information resources. 
<URL: http://www.vuw.ac.nz/dlis/courses/847/m2locevl.html> [Accessed: 29 Nov. 1996]

White, M.D. and Abels, E.G., 1995, Measuring service quality in special libraries: lessons from service
marketing. Special Libraries, Vol. 86, no. 1, Winter, pp. 36-45.

Wiedner University, Wolfram Memorial Library, 1996, Teaching critical evaluation skills for World Wide Web resources.
<URL: http://www.science.widener.edu/~withers/webeval.htm> [Accessed: 4 Dec. 1996]

Zeithaml, V.A., Parasuraman, A. and Berry, L.L., 1990, Delivering quality service: balancing customer perceptions and expectations. New York: Free Press.

Zorn, P., Ermanoil, M., Marshall, L. and Panek, M., 1996, Advanced Web searching: tricks of the trade. Online, Vol. 20, no. 3, May/June, pp. 15-28.

Mailing lists:

BESTWEB		send email to listserv@vm.ege.edu.tr 
			subscribe BESTWEB <your name>

BI-L			send email to listserv@bingvmb.cc.binghamton.edu 
			subscribe BI-L <your name>

COLLIB-L		send email to listproc@willamette.edu 
			subscribe COLLIB-L <your name>

Info-Quality-L		send email to majordomo@coombs.anu.edu.au 
			subscribe info-quality-l <your email address>

WAIS archive database at: <URL: gopher://cheops.anu.edu.au/7waissrc%3a/Coombs-db/ANU-Info-Quality-L.src> [Accessed: 4 Dec. 1996]

Libref-L			send email to listserv@listserv.kent.edu 
			subscribe libref-l <your name>

Net-Happenings		send email to listserv@lists.internic.net 
			subscribe net-happenings

NETTRAIN		send email to listserv@ubvm.cc.buffalo.edu 
			subscribe NETTRAIN <your name>

Web4Lib			send email to listproc@library.berkeley.edu 
			subscribe web4lib <your name>

DESIRE: Peer Review Report	
Project Number:	RE 1004 (RE)		
Project Title:	DESIRE - Development of a European Service for Information on Research and Education		
Deliverable Number: 	D3.22		
Version Number	1.0 		
Deliverable Title:	Selection Criteria for Quality Controlled Information Gateways		
Review Method:	Report Reading		
Principal Reviewer:	Name	Dr T. Matthew CIOLEK	
	Address	Research School of Pacific & Asian Studies, Australian National University, Canberra ACT 0200, Australia	
	E-Mail	tmciolek@coombs.anu.edu.au	
	Telephone	+61 (0)6 249 0110	
	Fax	+61 (0)6 257 1893	
	Credentials	Head, Internet Publications Bureau,6 years of full-time work designing gateways and online information systems Editor of  6 Virtual Libraries and  3 online serials. CV - see http://coombs.anu.edu.au/Depts/CCU/ciolek-tm.html	
Summary:	Relevant	5         	
	State-of-Art	4	
	Meets Objectives	4	
	Clarity	3
PART II - 4 
PART III - between 1 and 2
Appendices -  between 4 and 5	
	Value to Users	PART II - 5
PART III - 3
Appendices - 5	
Specific Criticisms	1	Part III of the report needs substantial simplification, possibly
by shifting repetitive details of the methodology and procedure to an
additional appendix. Also, conclusions need to be emphasised (but not reiterated) more clearly.	
	2	Part 5.2 lacks the consistency. The Scope Criteria section has a 3
part, nested hierarchical structure, a scheme which  NOT followed by the subsequent Content Criteria Form Criteria Process Criteria
Collection Management Criteria sections.	
	3	The final model in its present format appears to be unusable. The
sheer number of evaluative questions (127) and sub-questions
(additional 127 items) makes it a precise, informed but clumsy and
labour-intensive tool.	
	4	5. Further work on the model is needed. 

(a) a study of the intra-evaluator consistency (does the same person
give similar reply to the same question referring to the same resource seen for the 2nd or 3rd time ?)

(b) a study of inter-evaluator consistency (does the model yield
similar results for the same set of resources if used by different
evaluators ?)

(c) a study of the advantages/disadvantages of the proposed model as
compared with other evaluative schemes. Certainly, only a demonstrably better tool should be used.	
	The responses in italics refer to changes which will be incorporated into subsequent revisions, especially to the list of selection criteria. Responses in non-italicised text have been incorporated into version 1.1 of this document.	
Developer Response:	1	Agreed. This part of the report has been substantially rewritten to separate the methodologies and evolution of the model and the selection criteria.	
	2	The scope criteria were originally presented differently to emphasise that they are qualitatively different from the other sets of criteria.  However, the format in which the list is presented will be re-assessed with a view to making it clearer and more consistent.	
	3	In its present form the list is comprehensive and is intended to be used to provide a tool from which subject services can generate their own specific lists of criteria. The simplification of the list and an examination of weighting will result in a much more useable tool	
	4(a)	Intra-evaluator consistency 

Reliance on human selection has its strengths and weaknesses and it is acknowledged that for any evaluator the stringency of application of any set of criteria is variable.

We need to make sure that there is an acceptable level of reliability and consistency in the decisions made. Decisions made about the SCOPE criteria are likely to be close to 100% consistent, as these criteria and black and white.  Decisions based on the other criteria are likely to be less reliable as the evaluation involved is more complex.  I think further work is not viable in DESIRE 1 

KB, Lund and SOSIG selection staff will be working on distilling the model to create agreed working criteria for the three gateways this Summer - and hope to  highlight the most important criteria which should improve the consistency of decisions. This will be a necessary precursor to any future development work on weighting and scoring. (envisaged within the DESIRE II proposal).

We have now established discussion lists for European Section Editors and Correspondents, which provide a forum for reaching consensus on issues around the consistency of the selection processes in general and where appropriate individual resource selection. 	
	4(b)	Inter-evaluator consistency

When individual services distil the model, and have a more compact list of agreed working criteria with particular emphasis, then intra-and inter-consistency will improve.  

The model allows for formal review of the selection criteria employed, but in reality this should be achieved by open discussions about the selection criteria of a service. People can continually compare and contrast their decisions and reach and agreed understanding of the criteria. This is now happening via the discussion list which we have established.	
	4(c)	Advantages/disadvantages of the proposed model  as compared with other evaluative schemes. 

Currently we do not have resources to undertake such a study, though an informal comparison with a set of selection criteria developed by Alison Cooke ( as part of her PhD at Aberystwyth University) will be made.	
Other Reviewer	Name	Betsy Anagnostelis	
	Address	Medical Library, Royal Free Hospital School of Medicine
Rowland Hill Street,   London NW3 2PF,   United Kingdom	
	E-Mail	betsy@rfhsm.ac.uk	
	Telephone	+44 (0)171 830 2585	
	Fax	+44 (0)171 794 3534	
	Credentials	Librarian at the Royal Free Hospital School of Medicine, and has worked in medical libraries since 1990. She is also Assurance Officer for OMNI (Organising Medical Networked Information, a subject gateway funded as part of the eLib Programme to provide access to high quality biomedical networked information), and maintains a keen interest in all aspects of quality and evaluation of networked information.

As OMNI Assurance Officer, Betsy produced OMNI's Guidelines for Volunteers: Evaluating Resources for OMNI, and has continued to develop and adapt these as necessary. She also convenes the OMNI Advisory Group for Evaluation Criteria, which has been assessing the use of evaluation criteria by Internet review sites and evaluative subject gateways. Betsy has written evaluative reviews of electronic products and publications for Information World Review and Managing Information, and has given a number of presentations on issues of quality of information. 
She has authored or co-authored the following papers in the area of networked information quality (most recently reporting work of the OMNI Advisory Group for Evaluation Criteria):

Anagnostelis, Betsy. Filtering services and quality of information on the Internet, Inform 180, December 1995. p3.
Anagnostelis, Betsy and Cox, John. Data on the Internet: Evaluating the Quality or "Less is More". In C. J. Armstrong and R. J. Hartley (eds.)
ukolug96@warwick.ac.uk: UKOLUG State-of-the-Art Conference, Warwick, 17-19
July 1996, London: UKOLUG, 1996. pp59-69.
Cooke, Alison, McNab, Alison and Anagnostelis, Betsy. The good, the bad and the ugly: Internet review sites. In Online Information 96: the Proceedings of the 20th International Online Information Meeting, London,
3-5 December 1996, Oxford: Learned Information: pp33-40.
Anagnostelis, Betsy, Cooke, Alison and McNab, Alison. Thinking critically about information on the Web. Vine (forthcoming).	
Summary:	Relevant	5         (1 = poor, 5 = excellent)	
	State-of-Art	4.5	
	Meets Objectives	4.5	
	Clarity	4.5	
	Value to Users	5	
Specific Criticisms	1	2.1 Background to development of the model
Subject gateways consciously emphasise the importance of skilled human involvement in the assessment and 'quality control' of their selected Internet resources.  The core activity - selecting and attributing meaning to those resources is a human activity.  Subject gateways are run as academic services.

Not exclusively academic services? It is conceivable that
non-academically run SBIGs might develop?	
	2	4.1 Pre-test model
...
Root Definitions
A university owned and maintained system that selects and catalogues subject specialist Internet resources on the bases of quality and relevance, allowing structured access by a range of users in research and education...

What about practitioners? 	
	3	4.3 Pre-test list
...
The language used was standardised.  The criteria and attributes were phrased as a question.  The question format was chosen because it was the most common format found during the review, and because it reflects the evaluative nature of the selection process.

Some clear definitions could additionally have been provided?	
	4	Only one of the criteria was not used by any of the services.  This was the Special Needs criteria in the Scope section.  None of the services said their users had any special needs that would affect the resources that were selected (e.g. disabled users requiring large print or audio resources...

Nonetheless, I'm happier that:

It was therefore decided that none of the items should be removed from the list, as they might be appropriate for some services.	
	5	The testers' comments in answer to the open questions gave a consistent picture of the relative importance of the different categories of criteria. The scope criteria and content criteria tended to carry the most weight in the selection process of the majority of the services.  One service said the collection management criteria also carried most weight.  The process and form criteria tended to carry the least weight.

Weighting can possibly turn out to be a key concept when it comes to distinguishing which criteria apply to different disciplines. This, again, might be an area for further exploration.	
	6	Cost...  What if resources are under copyright?

Might copyright warrant a mention as a separate category? It's not strictly speaking cost related	
	7	Cataloguing Policy
Granularity

At what level will resources be selected/catalogued?
Will resources be considered at the web site/Usenet group level or the web page/Usenet article level?  

This defines almost how the questions might best be phrased, and their relevance (in the document, there is a preponderance of questions referring to "sites"); in the long term, it might be useful to distinguish among criteria that apply more to sites than to individual resources or collections of resources.	
	8	Resource description

What is the minimum amount of information needed to create a resource description in your catalogue?  I.e.  what basic information MUST a resource contain to be selected? (E.g. in a WWW document, contact details, last update details etc.)

Perhaps mention authorship as an example? As an illustrative example, it would be particularly apposite.	
	9	Authority and Reputation of the Source
Who provided the information?
Is the source attributable to a reputable author or organisation?

Is the URL a university server?

or other reputable organisation?	
	10	Has the information been filtered?

Is the site linked to by multiple Internet sites?

By other selective subject gateways? (Perhaps some discrimination of the quality of the linking site might be a good idea: not all services that might be considered to provide selectivity are perhaps of equal standing?)	
	11	Uniqueness

Perhaps worth making a specific mention of mirror sites? As a special case, maybe. Worth mentioning them in the scope notes, too? (Are they acceptable?)	
	12	Currency and Adequacy of Maintenance (See 'Resource Integrity' section for details of this section)

 I would agree that Currency is a feature of a resource, and should be
accommodated in this section. I would argue that Adequacy of Maintenance is information provider related and can slip out of this section and fit more appropriately in the 'Resource Integrity' section. Perhaps the best solution is to separate the two concepts entirely?	
	13	Ease of Navigation

Is it easy to navigate the resource?

Does it take more than three 'clicks' (three links) to get to something
interesting?

To get to ‘substantive information’? ('Something interesting' is very
subjective; admittedly, 'substantive information' might be too restrictive. Perhaps define it as something 'relevant to the objective' of the service?)	
	14	Is it easy to search the resource?

Does the system have an effective search facility?
...
How effectively can information be retrieved from the resource?

I'm not sure I understand the difference between these two questions?	
	15	Information Integrity  (work of the Information Provider)
Is the information current and up to date?

If the site contains data or information that is time-sensitive, how current is this data and information?
How current is the material included in each update?
Is a date given stating when the web item was mounted?
Are time-sensitive resources available in near real-time?
Do the stated dates respond to the information in the resource?
Is the date given stating when the web item was created?

Is the information durable in nature?

How time-sensitive is the information, and how does this relate to frequency of update? (e.g. for resources such as timetables, schedules and conference announcements)
If it is a static resource (not updated) will the information be of lasting use to the audience?
Is the information of a type that has a limited period of use?

These two paragraphs could make up a new 'Currency' section under Content Criteria.	
	16	Appendix III: Quality / selection definitions, models and methods in use

This is a fascinating review of quality models, unravelling interesting possibilities. I'm unclear about how the evaluation criteria relate to this specifically, although as a broad outline I have no difficulty with it.	
	17	Appendix V: Selection criteria of other Internet services
1 Introduction
Nineteen  selective subject gateways

I have a little bit of difficulty with the use of the same descriptive
phrase 'selective subject gateways' in this latter context (Appendix V:) -
or else with the definition of a selective subject gateway as a university-based service primarily. Some critical work has been conducted in comparing the various types of services; I hesitate to suggest the publications, as I am co-author of all three (one forthcoming).
	
	18	Appendix VII
Conclusions
As noted in the introduction most of the surveys carried out were not designed to elicit feedback related to resource selection criteria...

While mentioned later on in Appendix VIII, this might also be an
appropriate place to mention Alison Cooke's work with OMNI. (Mentioned under: End-user perceptions of information quality in an network environment
One respondent noted that criteria currently used were not validated with reference to their suitability for the network environment or to the subject areas being covered by the database itself.  One of the subject gateways (OMNI) is conducting research into end user perceptions of information quality.) In fact a correction is due here, since Alison is conducting research in her own capacity - OMNI is simply providing the test bed.
	
	19	Appendix VIII: E-Mail questionnaire sent to subject gateways, July 1996.

Granularity
...The key question is: at what level should selection take place?
Individual Files? The Server?  This may not seem to be particularly relevant to quality selection issues, but quality guidelines may be different for different levels".

Yes - these match my comments above under Granularity. Interesting that they have been picked up through the email questionnaire...	
	20	Another respondent noted that there was a "lack of in-depth work to
formalise criteria according to different types of resources (e.g.
electronic journals versus image databases, etc.)".

Alison Cooke is now beginning to conduct research specifically into this aspect.	
	21	Increased co-operation with information providers
Several subject gateways noted the need for increased co-operation with the information providers themselves.  Some services already contact information providers to fill-in gaps in their knowledge about a resource (if for example a resource was missing a date) and would be prepared to contact them again to check whether resources were going to be updated regularly.  Others would like information providers to contact them with details of new or updated sites when necessary or to make some comments on the resource description made by the service. 

Indeed, this is something I would add to the model: draw an arrow from 'add to catalogue' to 'request/inform provider'?	
	22	End-user perceptions of information quality in an network environment
One respondent noted that criteria currently used were not validated with reference to their suitability for the network environment or to the subject areas being covered by the database itself.  One of the subject gateways (OMNI) is conducting research into end user perceptions of information quality.

See comments above about desirable correction here.	
	23	Finally, one more addition to the model: 'suggest alternative keywords to user or other sources to search’	
Developer Response:	The responses in italics refer to changes which will be incorporated into subsequent revisions, especially to the list of selection criteria. Responses in non-italicised text have been incorporated into version 1.1 of this document.	
	1	Change to ‘currently run as academic services’	
	2	The list of customers (who benefits) includes practitioners. The fundamental activity which practitioners would carry out was felt to be research.	
	3	Noted for the final criteria selection tools	
	4	Noted, but requires no action	
	5	Agreed. Weighting will be a key factor when individual services define their own selection criteria.  The next phase of the project involves developing the criteria to be used for the demonstrator gateway, and we will try and weight a selection of the criteria.  A case study will be written to describe how weighting can be applied in practice.	
	6	Copyright will be given a separate section in the final criteria list.	
	7	At what level....
It is acknowledged that creating lists of criteria that are specific to a particular category of networked resources (server/document/mail archive etc.) could be usefully developed.  (Indeed the reviewer points out later, that useful work is being done in this area by Alison Cooke of Aberystwyth University).	
	8	No additional examples will be given.	
	9	Change this to 'Is the URL a university server, or other reputable organisation?'	
	10	Has the information been filtered....
Agreed that in the final criteria list this will  be changed to: 'Has the information been through a reputable filter?'
Is the site linked to by multiple Internet sites....
Agreed.  In the final criteria list this will be changed to: 'Is the site linked to by any reputable sites?'	
	11	Agreed.  Two questions should be added in the tips and hints side of the table: ' Is the site a mirror or the original?' and 'Is the mirror regularly updated?'	
	12	Agreed:  I was always torn between whether to put the 'Information Integrity' section into 'Process' or 'Content' criteria.  Betsy's comments make me think we should have put them in 'Content' and leave the 'Process' criteria to include issues which are particular to networked information (content criteria apply to networked and print information alike). 	
	13	Change to read  'substantive information'	
	14	Agreed	
	15	It is agreed that these two sections could make up a new 'Currency' section under Content Criteria	
	16	This section was included as background to the problematic concept of quality and to emphasise that definitions of quality are themselves changing rapidly over time.	
	17	Change to read ‘selective Internet services’	
	18	Amend to read: ‘As noted in the introduction most of the surveys were not designed to elicit feedback related to resource selection criteria (although OMNI have work in progress with Alison Cooke of Aberystwyth University on end-user perceptions of information quality in the networked environment).  However, some of the existing results may help indirectly..........’	
	19	Noted	
	20	Noted	
	21	This will be incorporated into the model	
	22	Noted	
	23	The model will be amended to cover processes which follow on from failed searches	

The reviewers considered the report interesting, ambitious, worthwhile and badly needed piece of methodological work. Betsy Anagnostelis felt that it would become a classic in its field, and a standard 
reference, if promoted. The Appendix materials were considered very thorough compilations and provide useful supportive documentation for the report. The strengths of the report are particularly that it summarises an enormous amount of information; and also that it does so in a usable manner. While it contains no earth-shattering new developments, it will be useful for anyone contemplating setting up a 'selective subject gateway'. 

The report was intended to be a comprehensive, systematic and up to date review of current approaches to the selection process. The peer review indicates, at this early stage, that we have produced the necessary groundwork from which widely disseminated useful tools might evolve within the project timescales.