Intelligent Searching Agents on the Web

Many web search engines use the concept of a'spider' - automated software which goes out onto the web and trawls through the contents of each server it encounters, indexing documents as it finds them. This approach results in the kinds of databases maintained by services such as Alta Vista and Excite - huge indexes to a vast chunk of what's currently available on the web. However, the problems which users can face when using such databases are beginning to be well documented. A recent JISC-funded investigation [1] into the use of web search engines indicates that users can typically encounter a number of difficulties. These include the issue of finding information relevant to their needs, and the problem of information overload - when far too much information is returned from a search.