OAI collection description and set semantics

Caroline Arms, Andy Powell, Mogens Sandfaer

Introduction

Each OAI repository can be thought of as a collection - a collection of metadata records and/or "full content" items. The collection of records and/or items may, optionally, be partitioned into one or more sub-collections, known as sets. It should be noted that OAI repositories need not contain "full content" items. For example, a subject gateway (a database of metadata about remote Internet resources) is a repository containing a collection of records but no items. A pre-print archive is an example of a repository that contains a collection of items and a collection of records.

This document considers the description of OAI repositories and sets and makes recommendations for mechanisms to encode such collection descriptions within the OAI Metadata Harvesting Protocol (OAI-MHP). The intention is to develop a generic collection description mechanism that can be used across all OAI repositories, i.e. one that is applicable to all domains, and that is rich enough to support:

• discovery of repositories and/or sets by end-users,
• automated selection of repositories and/or sets by OAI service providers.

Ideally, the descriptions should be simple enough that they will be provided by the majoritory of repositories.

This document does not consider the more general issues of collection description, nor does it consider how the OAI-MHP can be used to harvest collection description metadata about arbitrary external collections, though that is a perfectly valid use of the protocol.

Terminology

repository

a collection of metadata records and/or full content items that can be accessed using the OAI-MHP.

set

a sub-collection of metadata records and/or full content items within a repository.

item

a "full content" resource available from a repository.

record

a description (metadata) available from a repository.

Existing collection description within the OAI-MHP

There is one existing collection description mechanism within the OAI-MHP. The response to an Identify request may contain a list of description containers, which provide an extensible mechanism for communities to describe their repositories.

The 1.1 specification contains two sample XML Schemas - oai-identifier and eprints:

• The oai-identifier may be used by repositories that choose to comply with a specific format of unique identifiers for records.
• The eprints container has been agreed upon by the OAI e-print community. The container is used to describe content and policy properties of repositories in the e-print community.

It should be noted that the oai-identifier schema doesn't provide a true collection description, it merely describes the format of identifiers used by the repository. The eprint schema provides a collection description covering both the items (dataPolicy) and the records (metadataPolicy). However, the eprint schema is fairly minimal. For example, there is no indication of the subject coverage of the repository.

In general, responses to the Identify request describe the repository as a whole - there is no agreed mechanism for separately describing the collection of items and the collection of records within the same repository.

The current version of the protocol provides no mechanism for describing the sets within a repository, other then providing the setName as part of the response to a ListSets request.

Current usage of sets

An analysis of the usage of sets by currently registered OAI repositories is available in Appendix A.

Collection description schemas

In order to share descriptions of repositories and sets a collection description schema needs to be agreed. Five possibilies are suggested here:

• Dublin Core - the 15 DCMES elements (unqualified).
  <http://www.dublincore.org/documents/dces/>
• Encoded Archival Description (EAD) - a standard for encoding archival finding aids.
  <http://www.loc.gov/ead/>
• The eprints schema - use the existing eprints schema, or extend the existing schema to make it more applicable to non-eprints archives.
  <http://www.openarchives.org/OAI_protocol/openarchivesprotocol.html#eprint>
• RSLP collection description schema - an application of qualified Dublin Core built on an underlying model of collections and their descriptions.
  <http://www.ukoln.ac.uk/metadata/rslp/schema/>
• UDDI/WSDL - The Universal Description Discovery and Integration specification provides a mechanism for describing services on the Internet.
  <http://www.uddi.org/>

Of these, there is some benefit in using simple Dublin Core because of its use elsewhere in the protocol. EAD has a strictly archival background and may not be applicable across the whole range of OAI repositories. The RSLP schema may be considered to be over-complex for use within OAI. The eprints schema may not be considered complex enough. UDDI primarily focuses on 'service' description, though it may also provide a framework for describing collections.

It is worth noting that mappings between the DC, EAD and RSLP schemas already exist.

Possible protocol enhancements

As described above, the OAI-MHP already provides a mechanism for describing the repository as a whole, using the description container within the response to an Identify request. All of the schemas listed above could be encoded within the description container, provided a suitable XML schema is made available.

To provide descriptions of the sets within a repository the protocol will need to be enhanced. For example, it would be possible to add an optional setDescription container within the response to a ListSets request. A response might look like this:

<?xml version="1.0" encoding="UTF-8"?>
<ListSets xmlns="http://www.openarchives.org/OAI/1.1/OAI_ListSets"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://www.openarchives.org/OAI/1.1/OAI_ListSets
                              http://www.openarchives.org/OAI/1.1/OAI_ListSets.xsd">
  <responseDate>2001-06-01T19:20:30-04:00</responseDate> 
  <requestURL>http://an.oa.org/OAI-script?verb=ListSets</requestURL>
  <set>
    <setSpec>Oceanside</setSpec>
    <setName>Oceanside University of Nebraska</setName>
    <setDescription>
      <dc xmlns="http://purl.org/dc/elements/1.1/" 
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
          xsi:schemaLocation="http://purl.org/dc/elements/1.1/
                              http://www.openarchives.org/OAI/1.1/dc.xsd">
        <subject>Marine biology</subject>
        <publisher>University of Nebraska</publisher>
        <rights>Metadata may be used without restrictions as long as the oai
        identifier remains attached to it.</rights>
       </dc>
    </setDescription>
  </set>
</ListSets>

However, it is possible to view each repository and/or set as two collections - a collection of items and a collection of metadata records. It should be noted that, in the discussion above, the use of description and setDescription containers does not provide a mechanism for separately describing these two collections within the repository and/or set. In order to support separate descriptions, separate containers would be required. For example: itemsDescription, recordsDescription, setItemsDescription and setRecordsDescription.

Rather than providing collection descriptions in-line within the protocol, an alternative approach might be to provide some mechanism for linking to an external collection description about the repository and/or sets.

Summary of issues

It is recommended that further work be carried out to propose answers to the following questions:

• Does the OAI-MHP need an agreed mechanism for the description of repositories and sets?
• If so, what functionality should such descriptions support, i.e. what level of detail is required in the descriptions? Note that the focus here should be on service provider requirements, bearing in mind the cost of creating collection descriptions at the repository.
• If it is agreed that sets need to be described separately from the whole repository, then how should the protocol be enhanced to exchange such descriptions? Should the response to the ListSets command be extended as outlined above, or should the response to the Identify request contain information about both repository and sets?
• Do the collections of items and collections of records need describing separately? If so, the collection description schema will need to be rich enough to describe both types of collections and the protocol may need enhancing to separately identify the collection of items and the collection of records.

Appendix A - Analysis of set usage in registered OAI repositories

By Caroline Arms, Library of Congress
As of 2001-10-22