Large scale acquisition and maintenance from the web without source access

Dome may be used to extract knowledge from web sites in such a way that the extraction process may be reliably replayed, and facilities are provided to batch process a large collection of pages in one operation.

Abstract

Although di#erent web sites structure their pages di#erently, the pages within a single site are often generated from a database and have a regular layout from which it is possible to extract information automatically. Dome is a visual tool for manipulating tree-structured documents. It can import and export in XML or HTML formats, making it ideal for harvesting information from web pages. Editing is performed using a direct manipulation interface and the operations are recorded for later playback. The knowledge extracted from a web page may be updated by replaying the recorded sequence when the source page changes. The same sequence can be applied to other pages with a similar format, and facilities are provided to batch process a large collection of pages in one operation. In this paper we describe how Dome may be used to extract knowledge from web sites in such a way that the extraction process may be reliably replayed. Keywords Knowledge acquisition tools, Programming by demonstration systems, Programming by example, Visual languages, XML editors