8 Future Developments

This section describes options for future developments of a community-based Web-monitoring and auditing system such as WebWatch.

The Business Case

The WebWatch project has provided useful information on the development of Web sites within a number of UK communities. An indication of the usefulness of a project such as WebWatch can be gauged by observing the development of commercial services which have a similar function to Web Watch.

NetMechanic (their home page is shown in Figure 8-1) provide a free service for reporting on single Web pages. A number of commercial packages are also available which currently cost between $24 per month and $399 per year for reporting on a single Web page.

Figure 8-1 NetMechanic Home Page

NetMechanic is available at the address <URL: http://www.netmechanic.com/ >

WebSiteGarage (their home page is shown in Figure 8-2) also provide a free service for reporting on single Web pages. A number of commercial packages are also available which currently cost between $24 per month and $399 per year for reporting on a single Web page.

Figure 8-2 WebSiteGarageHome Page

WebSiteGarage is available at the address <URL: http://www.websitegarage.com/ >


Possible technological developments for a WebWatch service are given below.

Database Access to WebWatch Data

One of the main limitations to development of the WebWatch project is the need to develop ad hoc Unix scripts in order to process the data. In addition, analysis of the data can only be carried out by staff involved with the WebWatch project. It would be useful to store the WebWatch data in a backend database and to provide access to the data using a Web browser.

Evaluation of Web Site Accessibility

The WebWatch software has a possible role in the evaluation of the accessibility of a Web site. For example, WebWatch could report on the occurrence of HTML features which cause accessibility problems (e.g. missing ALT attributes on <IMG> elements). The WebWatch project could work in conjunction with projects such as DISinHE, who have a remit to support the use of accessible Web sites and the W3C WAI (Web Accessibility Initiative).

Note that the WebWatch software is included in the Web Accessibility Initiative list of tools at
<URL: http://www.w3.org/WAI/ER/existingtools.html >.

Monitoring SSL Deployment

WebWatch could be used to monitor deployment of SSL. SSL (Secure Sockets Layer) is used to provide access to secure Web services. Note that a commercial SSL server survey is also available from Netcraft, at a price of £1,200 for a monthly updated analysis reflecting the topology of encrypted transactions electronic commerce on the Internet - see <URL: http://www.netcraft.com/ssl/ >.

Use of Harvest/NG

During the development of the WebWatch robot links were made with the developers of the Harvest software. The WebWatch project team have provided input into the future development of Harvest, which is known as Harvest/NG. It is now recommended that future WebWatch work should make use of the Harvest/NG software.

Development of a Cachability Checking Service

The WebWatch project has developed a Web-based HTTP analysis service. In order to enable information providers to check whether a Web resource can be cached, it would be useful to develop a Cachability checking service. Although this can be done very easily for HTTP/1.0 servers, for HTTP/1.1 servers it would be necessary to develop a more sophisticated service, it order to check the status of the Etag field.

Configurable User-Agent Field

Web sites are beginning to provide different content depending on the user-agent (browser) which accesses them. It would be useful if it was possible to configure the WebWatch robot software so that it could provide reports on services which make use of user-agent negotiation.


Regular WebWatch trawls of communities such as the main entry points for higher education and other public sector services would be valuable. This would enable trends such as the use of server software (as monitored globally by Netcraft which is shown at <URL: http://www.netcraft.com/Survey/ >) to be carried out.

A comparison of UK communities with other communities (e.g. UK University entry points with US University entry points, UK University Web sites with commercial Web sites) would also be useful.