Selection criteria for quality controlled information gateways
Work Package 3 of Telematics for Research project DESIRE (RE 1004)
Title page
Table of Contents

Previous - Next

DESIRE: Peer Review Report
Project Number:RE 1004 (RE)
Project Title:DESIRE - Development of a European Service for Information on Research and Education
Deliverable Number: D3.22
Version Number1.0
Deliverable Title:Selection Criteria for Quality Controlled Information Gateways
Review Method:Report Reading
Principal Reviewer:Name Dr T. Matthew CIOLEK
AddressResearch School of Pacific & Asian Studies, Australian National University, Canberra ACT 0200, Australia
E-Mailtmciolek@coombs.anu.edu.au
Telephone+61 (0)6 249 0110
Fax+61 (0)6 257 1893
CredentialsHead, Internet Publications Bureau,6 years of full-time work designing gateways and online information systems Editor of 6 Virtual Libraries and 3 online serials. CV - see http://coombs.anu.edu.au/Depts/CCU/ciolek-tm.html
Summary:Relevant 5
State-of-Art4
Meets Objectives4
Clarity3

PART II - 4

PART III - between 1 and 2

Appendices - between 4 and 5

Value to UsersPART II - 5

PART III - 3

Appendices - 5

Specific Criticisms1 Part III of the report needs substantial simplification, possibly

by shifting repetitive details of the methodology and procedure to an

additional appendix. Also, conclusions need to be emphasised (but not reiterated) more clearly.

2Part 5.2 lacks the consistency. The Scope Criteria section has a 3

part, nested hierarchical structure, a scheme which NOT followed by the subsequent Content Criteria Form Criteria Process Criteria

Collection Management Criteria sections.

3The final model in its present format appears to be unusable. The

sheer number of evaluative questions (127) and sub-questions

(additional 127 items) makes it a precise, informed but clumsy and

labour-intensive tool.

45. Further work on the model is needed.

(a) a study of the intra-evaluator consistency (does the same person

give similar reply to the same question referring to the same resource seen for the 2nd or 3rd time ?)

(b) a study of inter-evaluator consistency (does the model yield

similar results for the same set of resources if used by different

evaluators ?)

(c) a study of the advantages/disadvantages of the proposed model as

compared with other evaluative schemes. Certainly, only a demonstrably better tool should be used.

The responses in italics refer to changes which will be incorporated into subsequent revisions, especially to the list of selection criteria. Responses in non-italicised text have been incorporated into version 1.1 of this document.
Developer Response:1 Agreed. This part of the report has been substantially rewritten to separate the methodologies and evolution of the model and the selection criteria.
2The scope criteria were originally presented differently to emphasise that they are qualitatively different from the other sets of criteria. However, the format in which the list is presented will be re-assessed with a view to making it clearer and more consistent.
3In its present form the list is comprehensive and is intended to be used to provide a tool from which subject services can generate their own specific lists of criteria. The simplification of the list and an examination of weighting will result in a much more useable tool
4(a)Intra-evaluator consistency

Reliance on human selection has its strengths and weaknesses and it is acknowledged that for any evaluator the stringency of application of any set of criteria is variable.

We need to make sure that there is an acceptable level of reliability and consistency in the decisions made. Decisions made about the SCOPE criteria are likely to be close to 100% consistent, as these criteria and black and white. Decisions based on the other criteria are likely to be less reliable as the evaluation involved is more complex. I think further work is not viable in DESIRE 1

KB, Lund and SOSIG selection staff will be working on distilling the model to create agreed working criteria for the three gateways this Summer - and hope to highlight the most important criteria which should improve the consistency of decisions. This will be a necessary precursor to any future development work on weighting and scoring. (envisaged within the DESIRE II proposal).

We have now established discussion lists for European Section Editors and Correspondents, which provide a forum for reaching consensus on issues around the consistency of the selection processes in general and where appropriate individual resource selection.

4(b)Inter-evaluator consistency

When individual services distil the model, and have a more compact list of agreed working criteria with particular emphasis, then intra-and inter-consistency will improve.

The model allows for formal review of the selection criteria employed, but in reality this should be achieved by open discussions about the selection criteria of a service. People can continually compare and contrast their decisions and reach and agreed understanding of the criteria. This is now happening via the discussion list which we have established.

4(c)Advantages/disadvantages of the proposed model as compared with other evaluative schemes.

Currently we do not have resources to undertake such a study, though an informal comparison with a set of selection criteria developed by Alison Cooke ( as part of her PhD at Aberystwyth University) will be made.

Other ReviewerName Betsy Anagnostelis
AddressMedical Library, Royal Free Hospital School of Medicine

Rowland Hill Street, London NW3 2PF, United Kingdom

E-Mailbetsy@rfhsm.ac.uk
Telephone+44 (0)171 830 2585
Fax+44 (0)171 794 3534
CredentialsLibrarian at the Royal Free Hospital School of Medicine, and has worked in medical libraries since 1990. She is also Assurance Officer for OMNI (Organising Medical Networked Information, a subject gateway funded as part of the eLib Programme to provide access to high quality biomedical networked information), and maintains a keen interest in all aspects of quality and evaluation of networked information.

As OMNI Assurance Officer, Betsy produced OMNI's Guidelines for Volunteers: Evaluating Resources for OMNI, and has continued to develop and adapt these as necessary. She also convenes the OMNI Advisory Group for Evaluation Criteria, which has been assessing the use of evaluation criteria by Internet review sites and evaluative subject gateways. Betsy has written evaluative reviews of electronic products and publications for Information World Review and Managing Information, and has given a number of presentations on issues of quality of information.

She has authored or co-authored the following papers in the area of networked information quality (most recently reporting work of the OMNI Advisory Group for Evaluation Criteria):

Anagnostelis, Betsy. Filtering services and quality of information on the Internet, Inform 180, December 1995. p3.

Anagnostelis, Betsy and Cox, John. Data on the Internet: Evaluating the Quality or "Less is More". In C. J. Armstrong and R. J. Hartley (eds.)

ukolug96@warwick.ac.uk: UKOLUG State-of-the-Art Conference, Warwick, 17-19

July 1996, London: UKOLUG, 1996. pp59-69.

Cooke, Alison, McNab, Alison and Anagnostelis, Betsy. The good, the bad and the ugly: Internet review sites. In Online Information 96: the Proceedings of the 20th International Online Information Meeting, London,

3-5 December 1996, Oxford: Learned Information: pp33-40.

Anagnostelis, Betsy, Cooke, Alison and McNab, Alison. Thinking critically about information on the Web. Vine (forthcoming).

Summary:Relevant 5 (1 = poor, 5 = excellent)
State-of-Art4.5
Meets Objectives4.5
Clarity4.5
Value to Users5
Specific Criticisms1 2.1 Background to development of the model

Subject gateways consciously emphasise the importance of skilled human involvement in the assessment and 'quality control' of their selected Internet resources. The core activity - selecting and attributing meaning to those resources is a human activity. Subject gateways are run as academic services.

Not exclusively academic services? It is conceivable that

non-academically run SBIGs might develop?

24.1 Pre-test model

...

Root Definitions

A university owned and maintained system that selects and catalogues subject specialist Internet resources on the bases of quality and relevance, allowing structured access by a range of users in research and education...

What about practitioners?

34.3 Pre-test list

...

The language used was standardised. The criteria and attributes were phrased as a question. The question format was chosen because it was the most common format found during the review, and because it reflects the evaluative nature of the selection process.

Some clear definitions could additionally have been provided?

4Only one of the criteria was not used by any of the services. This was the Special Needs criteria in the Scope section. None of the services said their users had any special needs that would affect the resources that were selected (e.g. disabled users requiring large print or audio resources...

Nonetheless, I'm happier that:

It was therefore decided that none of the items should be removed from the list, as they might be appropriate for some services.

5The testers' comments in answer to the open questions gave a consistent picture of the relative importance of the different categories of criteria. The scope criteria and content criteria tended to carry the most weight in the selection process of the majority of the services. One service said the collection management criteria also carried most weight. The process and form criteria tended to carry the least weight.

Weighting can possibly turn out to be a key concept when it comes to distinguishing which criteria apply to different disciplines. This, again, might be an area for further exploration.

6Cost... What if resources are under copyright?

Might copyright warrant a mention as a separate category? It's not strictly speaking cost related

7Cataloguing Policy

Granularity

At what level will resources be selected/catalogued?

Will resources be considered at the web site/Usenet group level or the web page/Usenet article level?

This defines almost how the questions might best be phrased, and their relevance (in the document, there is a preponderance of questions referring to "sites"); in the long term, it might be useful to distinguish among criteria that apply more to sites than to individual resources or collections of resources.

8Resource description

What is the minimum amount of information needed to create a resource description in your catalogue? I.e. what basic information MUST a resource contain to be selected? (E.g. in a WWW document, contact details, last update details etc.)

Perhaps mention authorship as an example? As an illustrative example, it would be particularly apposite.

9Authority and Reputation of the Source

Who provided the information?

Is the source attributable to a reputable author or organisation?

Is the URL a university server?

or other reputable organisation?

10Has the information been filtered?

Is the site linked to by multiple Internet sites?

By other selective subject gateways? (Perhaps some discrimination of the quality of the linking site might be a good idea: not all services that might be considered to provide selectivity are perhaps of equal standing?)

11Uniqueness

Perhaps worth making a specific mention of mirror sites? As a special case, maybe. Worth mentioning them in the scope notes, too? (Are they acceptable?)

12Currency and Adequacy of Maintenance (See 'Resource Integrity' section for details of this section)

I would agree that Currency is a feature of a resource, and should be

accommodated in this section. I would argue that Adequacy of Maintenance is information provider related and can slip out of this section and fit more appropriately in the 'Resource Integrity' section. Perhaps the best solution is to separate the two concepts entirely?

13Ease of Navigation

Is it easy to navigate the resource?

Does it take more than three 'clicks' (three links) to get to something

interesting?

To get to 'substantive information'? ('Something interesting' is very

subjective; admittedly, 'substantive information' might be too restrictive. Perhaps define it as something 'relevant to the objective' of the service?)

14Is it easy to search the resource?

Does the system have an effective search facility?

...

How effectively can information be retrieved from the resource?

I'm not sure I understand the difference between these two questions?

15Information Integrity (work of the Information Provider)

Is the information current and up to date?

If the site contains data or information that is time-sensitive, how current is this data and information?

How current is the material included in each update?

Is a date given stating when the web item was mounted?

Are time-sensitive resources available in near real-time?

Do the stated dates respond to the information in the resource?

Is the date given stating when the web item was created?

Is the information durable in nature?

How time-sensitive is the information, and how does this relate to frequency of update? (e.g. for resources such as timetables, schedules and conference announcements)

If it is a static resource (not updated) will the information be of lasting use to the audience?

Is the information of a type that has a limited period of use?

These two paragraphs could make up a new 'Currency' section under Content Criteria.

16Appendix III: Quality / selection definitions, models and methods in use

This is a fascinating review of quality models, unravelling interesting possibilities. I'm unclear about how the evaluation criteria relate to this specifically, although as a broad outline I have no difficulty with it.

17Appendix V: Selection criteria of other Internet services

1 Introduction

Nineteen selective subject gateways

I have a little bit of difficulty with the use of the same descriptive

phrase 'selective subject gateways' in this latter context (Appendix V:) -

or else with the definition of a selective subject gateway as a university-based service primarily. Some critical work has been conducted in comparing the various types of services; I hesitate to suggest the publications, as I am co-author of all three (one forthcoming).

18Appendix VII

Conclusions

As noted in the introduction most of the surveys carried out were not designed to elicit feedback related to resource selection criteria...

While mentioned later on in Appendix VIII, this might also be an

appropriate place to mention Alison Cooke's work with OMNI. (Mentioned under: End-user perceptions of information quality in an network environment

One respondent noted that criteria currently used were not validated with reference to their suitability for the network environment or to the subject areas being covered by the database itself. One of the subject gateways (OMNI) is conducting research into end user perceptions of information quality.) In fact a correction is due here, since Alison is conducting research in her own capacity - OMNI is simply providing the test bed.

19Appendix VIII: E-Mail questionnaire sent to subject gateways, July 1996.

Granularity

...The key question is: at what level should selection take place?

Individual Files? The Server? This may not seem to be particularly relevant to quality selection issues, but quality guidelines may be different for different levels".

Yes - these match my comments above under Granularity. Interesting that they have been picked up through the email questionnaire...

20Another respondent noted that there was a "lack of in-depth work to

formalise criteria according to different types of resources (e.g.

electronic journals versus image databases, etc.)".

Alison Cooke is now beginning to conduct research specifically into this aspect.

21Increased co-operation with information providers

Several subject gateways noted the need for increased co-operation with the information providers themselves. Some services already contact information providers to fill-in gaps in their knowledge about a resource (if for example a resource was missing a date) and would be prepared to contact them again to check whether resources were going to be updated regularly. Others would like information providers to contact them with details of new or updated sites when necessary or to make some comments on the resource description made by the service.

Indeed, this is something I would add to the model: draw an arrow from 'add to catalogue' to 'request/inform provider'?

22End-user perceptions of information quality in an network environment

One respondent noted that criteria currently used were not validated with reference to their suitability for the network environment or to the subject areas being covered by the database itself. One of the subject gateways (OMNI) is conducting research into end user perceptions of information quality.

See comments above about desirable correction here.

23Finally, one more addition to the model: 'suggest alternative keywords to user or other sources to search'
Developer Response:The responses in italics refer to changes which will be incorporated into subsequent revisions, especially to the list of selection criteria. Responses in non-italicised text have been incorporated into version 1.1 of this document.
1Change to 'currently run as academic services'
2The list of customers (who benefits) includes practitioners. The fundamental activity which practitioners would carry out was felt to be research.
3Noted for the final criteria selection tools
4Noted, but requires no action
5Agreed. Weighting will be a key factor when individual services define their own selection criteria. The next phase of the project involves developing the criteria to be used for the demonstrator gateway, and we will try and weight a selection of the criteria. A case study will be written to describe how weighting can be applied in practice.
6Copyright will be given a separate section in the final criteria list.
7At what level....

It is acknowledged that creating lists of criteria that are specific to a particular category of networked resources (server/document/mail archive etc.) could be usefully developed. (Indeed the reviewer points out later, that useful work is being done in this area by Alison Cooke of Aberystwyth University).

8No additional examples will be given.
9Change this to 'Is the URL a university server, or other reputable organisation?'
10Has the information been filtered....

Agreed that in the final criteria list this will be changed to: 'Has the information been through a reputable filter?'

Is the site linked to by multiple Internet sites....

Agreed. In the final criteria list this will be changed to: 'Is the site linked to by any reputable sites?'

11Agreed. Two questions should be added in the tips and hints side of the table: ' Is the site a mirror or the original?' and 'Is the mirror regularly updated?'
12Agreed: I was always torn between whether to put the 'Information Integrity' section into 'Process' or 'Content' criteria. Betsy's comments make me think we should have put them in 'Content' and leave the 'Process' criteria to include issues which are particular to networked information (content criteria apply to networked and print information alike).
13Change to read 'substantive information'
14Agreed
15It is agreed that these two sections could make up a new 'Currency' section under Content Criteria
16This section was included as background to the problematic concept of quality and to emphasise that definitions of quality are themselves changing rapidly over time.
17Change to read 'selective Internet services'
18Amend to read: 'As noted in the introduction most of the surveys were not designed to elicit feedback related to resource selection criteria (although OMNI have work in progress with Alison Cooke of Aberystwyth University on end-user perceptions of information quality in the networked environment). However, some of the existing results may help indirectly..........'
19Noted
20Noted
21This will be incorporated into the model
22Noted
23The model will be amended to cover processes which follow on from failed searches

The reviewers considered the report interesting, ambitious, worthwhile and badly needed piece of methodological work. Betsy Anagnostelis felt that it would become a classic in its field, and a standard reference, if promoted. The Appendix materials were considered very thorough compilations and provide useful supportive documentation for the report. The strengths of the report are particularly that it summarises an enormous amount of information; and also that it does so in a usable manner. While it contains no earth-shattering new developments, it will be useful for anyone contemplating setting up a 'selective subject gateway'.

The report was intended to be a comprehensive, systematic and up to date review of current approaches to the selection process. The peer review indicates, at this early stage, that we have produced the necessary groundwork from which widely disseminated useful tools might evolve within the project timescales.

Next Table of Contents


Page maintained by: UKOLN Metadata Group,
Last updated: 2-Apr-1998