EASTER (Evaluating Automated Subject Tools for Enhancing Retrieval)
For current discussions see also the EASTER
Project Blog.
Overview
Start date: 1 April 2009
End date: 1 September 2010
Funding: JISC
Information Environment Programme 2009-11
The purpose of the project is to:
- Test and evaluate existing tools for automated subject metadata generation;
- Better understand what is possible and what the limitations of current solutions are; and,
- Make subsequent recommendations for services employing subject metadata in the JISC community.
The rationale is based on the following arguments:
- Subject metadata are most important in resource discovery, yet most expensive to produce manually. There is a huge challenge facing UK HE digital collections, institutional repositories, and aggregators of institutional repository content, as to how to provide high quality subject metadata for increasing numbers of digital information at reasonable costs;
- Subject metadata are much more difficult to generate automatically especially in comparison to formal metadata such as file type, title, etc.; and,
- Due to the high cost of evaluation, automated subject metadata tools are rarely tested in live environments of use.
Selected tools for automated subject metadata generation will be tested in two contexts: by Intute cataloguers in the cataloguing workflow; and by end-users of Intute who search for information in Intute as part of their research, learning, and information management. The approach will comprise the following steps:
- First, a methodology for evaluating the tools will be developed;
- Selected tools will be evaluated using a created ‘gold standard’;
- The best tool(s) for the purposes of Intute will be implemented into a demonstrator that will feed its results into the cataloguing workflow; and,
- A task-based end-user retrieval study will be conducted to determine the contribution of automatically assigned terms and manually assigned terms, each alone and in combination, to retrieval success (retrieving relevant documents) and failure (missing relevant documents and retrieving irrelevant documents).
Project partners
- UKOLN, University of Bath (lead)
- Hypermedia Research Unit, University of Glamorgan
- Intute, MIMAS – University of Manchester
- Centre for HCI Design, City University London
- Dagobert Soergel (consulting expert)