Title:		Clumps Technology Overview
Written by: 	Ed Davidson
Version no:	V1.1	[26-Mar-97]	
Status:	Working Draft
Contact:	(work) edavidso@fdgroup.co.uk
		(home) ed@dolomite.demon.co.uk
============================================================

Table of Contents

Introduction	
Why clumps?	
What is a clump?	
Who creates, administers, and disseminates clump information?	
Clump Technology	
Basic Requirements	
How are clumps defined?	
Z39.50 Explain records	
Z39.50 Digital Collections Profile	
Dublin Core	
The Warwick Framework	
How is clump information accessed?	
Some clump scenarios:	
How is clump information referenced?	
URL: Universal Resource Location	
URN:  Universal Resource Name	
PURL: Persisent Universal Resource Location	
Authentication & authorisation & payments	
Implementation Issues	
Recommendations	
APPENDIX 1	
APPENDIX 2	
APPENDIX 3	

Introduction
This document is intended to provide an overview of the technological issues surrounding the implementation of Clumps.   Several possible types of implementation are discussed, and a list of related issues is presented along with some possible solutions.   This document does not provide a prescription for building Clumps, but rather, suggests some possible implementations and assumes that the individual Clump implementors will be best placed to provide a solution that most accurately matches their needs.   However, it is important to remember that one of the goals of the Clumping exercise is to provide interoperability between Clump implementations, and so it is vital that the various groups produce solutions that will inter-work with one another by the end of the project.

Why clumps?
As more and more networked services come on line, using different protocols to provide access to more and more varied types of data, problems of service discovery and navigation make themselves apparent.   How does an individual searcher know which services are available?  What type of data do they provide?  How is that data accessed? How much does it cost? Which specialisations are best provided for by which services?.   

One possible solution to these problems is to produce "clumps" of data services that share common features.  A set of databases may be "clumped" around a specific geographic location, or a subject specialisation, or an intellectual domain.  Clumps can therefore help to organise the content of the network and help searchers to select the right databases for the particular search they are engaged upon.     

Clumps may also represent networked services in their own right, for instance, in the library community,  a Clump of databases might be formed to provide a logical union catalogue, or to provide a regional Inter-library Loan service.   

What is a clump?

A clump may be no more than a  list of databases that share some common features like: 
	Regional location
	Content type
	Subject matter
	Domain type (archives, museums, libraries etc)
	Service type (ILL consortia, logical union catalogues, etc)

Clumps can be grouped into Physical and Virtual Clumps.  Physical Clumps would be a group of databases fronted by a single Target or Server, whilst Virtual Clumps are adhoc groups of databases that are accessed individually through several servers, but appear to the end uset as if they are a single entity.

Virtual clumps may only exist for the duration of a particular search session, and they may only exist within the user's search client software.   Most current Z39.50 clients allow the user to build up lists of known databases, so that for any particular search, the user can select a set of databases that seem appropriate to the search, and the client will search the selected list as if it were a single server and present the search results to the user.

Some "client-side" Clumping may be more permanent, in that the user may group commonly accessed databases under logical names, and then simply select the named group for a search session.   Popular clients such as ZNavigator or Willow already provide this kind of functionality.  One may even suggest that the classic Web Browsers also provide a similar concept with their hierarchical bookmark functions.

At the other end of the spectrum, Clumps based around a given service may represent Clumping at its most sophisticated.   Here the Clump is no longer simply a figment of software but a physical organisation, with administrative staff and a group of member service providers who collectively form the Clump.   End users of such a service may not care about which given databases actually provide the service, they are only interested in the services being offered. 

Who creates, administers, and disseminates clump information?

There seems to be three or four types of player in the Clump community who may provide some or all of the imformation required.
	The database owners and database access providers
	The clump owners or service providers
	The end user's host organisation
	Third party information gateway services or agencies
	A national agency

And there are at least three levels of descriptive metadata required to describe a Clump and its components:
•	The Clump itself
•	The databases which contribute to a Clump
•	The servers that provide access to the databases

The database owners are probably the best people to create metadata that describes their databases and servers.   Clump level metadata may come from the Clump owners themselves if it is a physical organisation otherwise Clump information may be provided by third-party agencies or data gateway providers at the end users site or at some data access service.

Some unanswered questions are:
	Should Clump and Database information be held centrally or should it be distributed throughout the network.   If it is to be held centrally then should there be a national centre, or should it be concentrated in several places (ie end user organisations, Clump providers, etc).  If it is to be distributed, then how does one locate it, and how does one navigate from one distributed Clump metadata site to another.  How are the various caches of Clump information kept in synch and up to date?

	Should Clump and Database information be officially registered, and if so by whom? how?
Clump Technology
Basic Requirements
	The architecture should use current technology and international standards wherever possible.
	The architecture should be fundamentally accessible to users of current Z39.50 and http client software.
	The architecture should be scalable, allowing for implementations ranging from basic “low-tech” pilot studies through to complex national structures.
	The architecture, should be suitable for library catalogue clumping, but also be applicable to the larger cross-domain search problem.
	The architecture should be open enough to allow for a variety of interoperable implementations.
	The architecture should allow for the definition of persistent clumps as well as allowing clients/users to define adhoc virtual clumps based on the results of searching metadata records describing individual databases.  This latter requirement allows users to discover databases that they did not previously know existed.
How are clumps defined?
In order to define a Clump and its constituent databases in way that will make sense to both human and mechanical readers, we are going to need network accessible metadata records that describe the clump and its components.  

There are at least three levels of description required:
	The Clump itself
	The constituent databases within a Clump
	The hosts or servers that provide access to the individual databases via well known protocols.

Clump Metadata needs to perform several functions.  Firstly it needs to describe the content and purpose of the clump in such a way that the end user can decide whether the Clump is useful to them or not, and secondly it needs to provide enough technical information to the end-users client software to enable it to make the necessary connections to the remote service.   The human-readable descriptive information needs to be indexible and searchable, so that end-users may search a database of Clump information in order to be returned a list of services that will best be able to satisfy the requirements of the search.

Such metadata systems are currently one of the hot topics on the Internet, and this means that there is much research into this area of information management, which also means that there are many possibly competing metadata systems under development and as yet no clearly winning solution.  So rather than mandate a Clumps metadata system, it would be much more pragmatic to look at some of the more promising systems under development.   The Clumps developers may then consider choosing one or more of these systems as the basis for storing and transferring Clump metadata.

The metadata formats examined in this paper are:
	Z39.50 Explain records
	Z39.50 Digital Collections Profile
	Dublin Core
	The Warwick Framework

Very simple listings of the data elements for each of these systems are included as Appendices at the end of this document.   Anyone interested in any of these systems should go to the source documents themselves, since the brief listings given here have been provided solely to provide some form of comparison between the systems under discussion.   

Z39.50 Explain records
<URL: http://lcweb.loc.gov/z3950/agency/1995doc.html>
Within the latest version of the Z39.50 Information Search and Retrieval protocol there is the facility for Z39.50 servers to mount a database of metadata describing the services and databases available from that server.   This metadata includes records that can be used to describe both actual databases and logical databases that consist of one or more physical databases.   Since this accords quite strongly with our concept of Clumps representing one or more physical databases these Explain database description records merit further examination.   This paper is not advocating the exclusive use of the Z39.50 Explain facility itself, but is suggesting that we might borrow Explain's database description record schema as a means of describing Clumps and their constituent databases, using whatever access protocol we think is most suitable.  A list of of the elements used in Explain databaseInfo records is given in Appendix 1 of this document.

Z39.50 Digital Collections Profile
<URL: http://lcweb.loc.gov/z3950/agency/profiles/collections.ps>
The Digital Collections Profile has been developed to provide a mechanism whereby hierarchical descriptions of museum or library collections can be navigated by a Z39.50 client in order to locate items or collections of interest to a particular search.  This profile is specifically aimed at the problem of cross-domain information retrieval, and at solving the problem of discovering which databases may be relevant to a given search.  This profile is assumed to provide the top-level cross-domain search capabilities, and assumes that other more detailed profiles will be brought into play as the searcher "drills down" into a particular information domain.

As with the Z39.50 Explain system, this paper is not advocating the adoption of Z39.50 for implementing clumps metadata systems.  However, this profile does appear to have been developed to solve a problem that is very similar to our own,
and it is therefore worthy of investigation by Clump implementors.  Appendix 3 has an outline listing of the Collection Description record elements used by this profile. 

Dublin Core
<URL: http://purl.org/metadata/dublin_core_elements.html>
The Dublin Core is the core set of descriptive elements that can describe a "document-like object".  The Dublin Core is still a topic of considerable debate, and although it might appear difficult to think of a Clump or a database as being a "document-like object", using Dublin Core for decribing Clumps has some distinct advantages:

The data elements are well known, and there are already  proposals for encoding Dublin Core in HTML, SGML, as well as mappings between Dublin Core and MARC, GRS-1, SUTRS, and other common records syntaxes.  This means that Dublin Core records can be accessed by a variety of network protocols, and the records themselves can be transferred between systems in a variety of encodings.  All of which bodes well for interoperability between implementations that use Dublin Core as the basis of their metadata records.

The Dublin Core elements along with Ashley Sanders's proposal for describing Clumps and Databases using these elements are listed in Appendix 2 of this document.

The Warwick Framework
<URL: http://cs-tr.cs.cornell.edu/Dienst/U1/2.0/Describe/ncstrl.cornell/TR96-1593>
The Warwick Framework is not really a metadata system in its own right, but is a system whereby different types of metadata may be packaged for transfer between systems.  Since we have at least three different levels of metadata, and a variety of metadata systems to choose from, it is quite possible that each descriptive level could be implemented using a different metadata system, in which case, an implementation of the Warwick Framework would be one way of packaging this information for transfer between systems (and between Clump implementations).

How is clump information accessed?
Although it is assumed that Clump database access will be via Z39.50, it is not so clear which protocol(s) will be used for accessing the metadata about Clumps.  There are a variety of information retrieval protocols that would suitable,  the most obvious ones being:
	Z39.50 / SR
	LDAP / X500
	Whois++
	HTTP
It is anticipated that a selection of these will be used in the pilot Clumps implementations.

Some clump scenarios:
Here are some possible architectures for connecting end users and their search clients to a variety of databases and Clumps.

1.	The Client searches a single target with a single database.  The end user has prior knowledge about the location and contents of the target and its database..

2.	The Client searches several targets, each mounting one or more databases. The Client keeps a local list of targets and databases internally or in a local server. This info may be gleaned by Explain or by external contact with database providers.  The Client may also be cacheing metadata provided to it by the remote targets.

3.	The Client communicates with a local or remote Search Broker that mediates between the client and the external servers and databases being accessed. The Search Brokers may provide additional services such as: locally cached clump/target/database information; the sorting and merging of retrieved metadata records into a single logical result set; user authentication and payment services and so on. 

Search Brokers may be mounted by the local users' site, or by third-party agencies, or in the cases where a clump is associated with an administrative organisation, then that organisation may mount its own broker.   In the above diagram, one of the Targets being accessed by the Search Broker may itself be another Search Broker.   

Search Brokers may be used to provide the distributed metadata storage and access mechanisms.  However navigational information concerning Clumps need not be confined to the Client or the Search  Brokers, it may be held in yet another set of directory servers or information systems.  Clumps metadata should therefore remain independent of any given protocol implementation in order to maximise interoperabilty.

How is clump information referenced?
This is question is really two related questions:
1.	How does one name or reference a link to an actual database in a way that will not force the updating of all clump records if the access details of the database change.
2.	How does one name or reference a link to other clump or database description records.

Here are some existing and emerging networked resource naming systems

URL: Universal Resource Location
This scheme identifies resources by providing details of their physical location on the network. URLs are by far the most common form of resource naming, but they are unreliable, since the resource's name changes as its physical location changes, and this is likely to cause major problems over time.

URL format is <protocol>://<machine address>/<filepath>/<filename>
URL example: http://lcweb.loc.gov/z3950/agency/

URN:  Universal Resource Name
More experimental than URLs URNs try to provide a universal name space that allows a variety of existing naming systems to be incorporated into a single format.  URNs rely on some third-party servers to resolve the permanent name into a temporary location, if the URL changes, then only the name resolvers need to be aware of this change, since the outside world will know the resource by its URN not its URL.

URN format is URN:<namespace>:<name>
URN example: urn:isbn:0124636379

PURL: Persisent Universal Resource Location
OCLC invented PURLs as a kind of halfway house between a URL and a URN.  Like URNs a PURL is a permenent name that gets resolved locally into the URL that actually points to the resource.  If the URL changes, only the PURL servers need to worry about this, the outside world only knows the resources by their PURLs.

PURL format is <protocol-to-access-resolver>://<resolver address>/<name>
PURL example: http://purl.oclc.org/home/fred/catalog

Authentication & authorisation & payments
This is potentially the most difficult aspect of the Clumps initiative.  Authentication and authorisation services may be provided by:
	End-User organisations (ie the Campus)
	Service provider organisation (Clump administrators)
	Database owners

They may use any one of several existing authentication systems such as Athens or Kerberos.

For the initial batch of pilot Clumps, it is assumed that access to resources will be free, but usage of resource (ie for ILL purposes etc) may be paid for.   While the JISC and others are trying to produce a national solution to authentication and authorisation, it may be that these issues need not be tackled in the initial development of Clumps.

Related to authentication and authorisation is the issue of invoicing and payments.  This  may be handled entirely electronically, or it may be conducted more conventionaly using paper invoices and normal offline financial transactions.   These issues are left entirely to the discretion of the individual Clump implementors.
Implementation Issues
This section recaps the major issues that need to be resolved by the Clump implementors:

1.	Selection of metadata schema definitions
2.	Selection of metadata access and transfer protocols
3.	Clump information access: centralised or distributed?
4.	End-user authentication and authorisation
5.	Invoicing and payments
6.	Inter-implementor co-ordination
7.	Inter-implementation interoperability
Recommendations
Since so many of these issues can only be answered by the implementors themselves, and since they each may well choose different solutions, it is very important that the different implementors keep in contact with one another, and are each aware of how the others are tackling the same problems.

There probably needs to be some kind of implementors' forum, perhaps in the form of workshops,  or some kind of co-ordinating comittee in order to ensure that, at the end of the day, the different Clump implementations are able to interoperate successfully with one another.

APPENDIX 1 
Z39.50 Explain records for database descriptions
This record can describe both a virtual database (or clump) and an actual database

Explain database description elements:
	Name of database
	Nicknames for this database name
	Pointer to external Explain server on another host
	Icon to be displayed when providing access to this database
	Fee		
	Availability	
	Title of database
	Keywords to find this description record
	News about this service
	Number of records in database
	Publisher contact information
	Supplier contact information
	Access Information
	List of other databases with common virtual parent (ie other clump members)
	List of sub databases 
		(ie list of actual databases that form this virtual database or clump)

APPENDIX 2
Dublin Core description elements:
	Identifier
	Relation
	Coverage
	Description
	Source
	Creator
	Subject
	Publisher
	Contributor
	Date
	Type
	Format
	Rights

May be used to describe clumps and databases thus:

ClumpInfo:
	Title
	Description
	Coverage
	Type
	Date
	Identifier
	Sources
	Keywords

DatabaseInfo:
	Title
	Description
	Coverage
	Type
	Sources
	Publisher
	Keywords

APPENDIX 3
Z39.50 Access To Digital Collections Profile

Digital Collections Descriptive Record:
Type of descriptive record
Brief description
Collection information:
	Collection name
	Databases (repeatable) (server and database name)
	Enumerated member:
		Brief description of member
		Pointer (database + record id or alternative identifier + type of 		identifier)
		What pointer points to
		Flag if fully enumerated
		Flag if children know this parent
Object information:
	Type of object
	Category of object
	Digital Object (repeatable)
		Actual digital object or pointer to digital object
		Flag if this is an authoratative reference
Associated description:
	Brief description of associated description
	Category of associated description
	Description (repeatable)
		Actual associated description or pointer to associated 			description
		Flag if this is an authoratative reference
Related Collection (repeatable):
	Collection:
		Collection name
		Server
		Database
	Relationship (repeatable)
	Relative level
	Description of relationship