Impact Analysis For Web Sites

Background

This briefing document provides advice on approaches to measuring the impact of a service provided by a Web site.

The document describes an approach to risk assessment and risk management which can allow the benefits of such services to be exploited, whilst minimising the risks and dangers of using such services.

Traditional Approaches To Impact Analysis

A traditional approach to measuring the impact of a Web site is to report on Web server usage log files [1]. Such data can provide information such as an indication of trends and growth in usage; how visitors arrived at the Web site; how users viewed pages on your Web site and details on the browser technologies used by your visitors.

However although such information can be useful, it is important to recognise that the underlying data and the data analysis techniques used may be flawed [2]. For example:

As search engines may deliver significant traffic to Web sites, potentially they may result in visitors who find the wrong type of information, and immediately leave after viewing the first page.
Use of caches, offline browsers, etc may influence the findings.
Trends may be misleading, as they may be influenced by changes in the Web site architecture (for example, if visitors spend more time on your Web site after a Web site redesign, does this indicate that they are finding additional useful resources, or wasting time looking for resources?

It should also be noted that care must be taken when aggregating usage statistics:

You should not aggregate monthly findings to determine, say, annual number of unique visitors.
Aggregation of data from disparate Web sites may give misleading information.

So although analysis of Web site usage data may be useful, the findings need to be carefully interpreted.

Other Approaches To Impact Analysis

Although Web site usage analysis may have flaws, there are other approaches which can be used to measure the impact of a Web site: Such alternative can be used to complement Web usage analysis.

Link analysis: If other Web sites have links to your Web site, this can be an indication of the value placed on you Web site. Services such as LinkPopularity.com [3] can provide such data. Keeping a record of the numbers of sites linking to you can also help show trends.
Analysis of social bookmarking services:: Services such as del.icio.us [4] allow you to bookmark resources. A useful aspect of the service is the ability to observe others who are bookmarking the same resource. So bookmarking your own Web site will allow you to record the numbers of people who bookmark your site. This may be a useful indicator, if the social bookmarking service you use if popular with your target audience.
User comments:: Comments from your user community can provide a particularly valuable way of measuring impact. Feedback can be obtained in a variety of ways: focus groups; online questionnaires, online guest books, etc.
Analysis of Web sites, mailing lists, Blogs, etc.:: Search engines such as Google, Technorati [5]], etc. may enable you to find comments about your Web site, but also provide various metrics which may be useful.

Possible disadvantages of using such services include:

Potential security and legal concerns.
Potential for data loss.
Reliance on third parties with whom there may be no contractual agreements.

Embedding Impact Analysis

In order to maximise the benefits, you may find it useful to develop an Impact Analysis Strategy. This should ensure that you are aware of the strengths and weaknesses of the approaches you plan to use, have mechanisms for gathering information in a consistent and effective manner and that appropriate tools and services are available.

References

Usage Statistics For Web Sites, QA Focus briefing document no. 84, UKOLN,
<http://www.ukoln.ac.uk/qa-focus/documents/briefings/briefing-84/>
Performance Indicators for Web Sites, B. Kelly, Exploit Interactive, issue 5, April 2000,
<http://www.exploit-lib.org/issue5/indicators/>
LinkPopularity.com,
<http://www.linkpopularity.com/>
del.icio.us,
<http://del.icio.us/>
Technorati,
<http://www.technorati.com/>