Learning page-independent heuristics for extracting data from Web pages
Computer NetworksPublished 1 May 1999
William W. Cohen, Wei Fan
Citations46
SJR quartileQ1
SJR score1.17
SNIP1.37
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A method for learning general, page-independent heuristics for extracting data from HTML documents that can substantially improve the performance of methods for learning page-specific wrappers.
Abstract
One bottleneck in implementing a system that intelligently queries the Web is developing `wrappers' — programs that extract data from Web pages. Here we describe a method for learning general, page-independent heuristics for extracting data from HTML documents. The input to our learning system is a set of working wrapper programs, paired with HTML pages they correctly wrap. The output is a general procedure for extracting data that works for many formats and many pages. In experiments with a collection of 84 constrained but realistic extraction problems, we demonstrate that 30% of the problems can be handled perfectly by learned extraction heuristics, and around 50% can be handled acceptably. We also demonstrate that learned page-independent extraction heuristics can substantially improve the performance of methods for learning page-specific wrappers.
Keywords
Computer Science
C4.5: Programs for Machine Learning
23,665 Citations1992J. R. Quinlan
A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.
Programs for Machine Learning
5,794 Citations1994Steven L. Salzberg, Alberto M. Segre
In his new book, C4.5: Programs for Machine Learning, Quinlan has put together a definitive, much needed description of his complete system, including the latest developments, which will be a welcome addition to the library of many researchers and students.
Elsevier eBooksFast Effective Rule Induction
3,767 Citations1995William W. Cohen
This paper evaluates the recently-proposed rule learning algorithm IREP on a large and diverse collection of benchmark problems, and proposes a number of modifications resulting in an algorithm RIPPERk that is very competitive with C4.5 and C 4.5rules with respect to error rates, but much more efficient on large samples.
Querying Heterogeneous Information Sources Using Source Descriptions
1,120 Citations1996Alon Y. Levy, Anand Rajaraman +1 more
The Information Manifold is described, an implemented system that provides uniform access to a heterogeneous collection of more than 100 information sources, many of them on the WWW, and algorithms that use the source descriptions to prune effciently the set of information sources for a given query are described.
Wrapper induction for information extraction
1,044 Citations1997Nicholas Kushmerick, Daniel S. Weld
This work introduces wrapper induction, a method for automatically constructing wrappers, and identifies hlrt, a wrapper class that is e(cid:14)ciently learnable, yet expressive enough to handle 48% of a recently surveyed sample of Internet resources.
Journal of Intelligent Information SystemsThe TSIMMIS Approach to Mediation: Data Models and Languages
832 Citations1997Héctor García-Molina, Yannis Papakonstantinou +6 more
TSIMMIS—The Stanford-IBM Manager of Multiple Information sources offers a datamodel and a common query language that are designed to support the combining of information from many different sources.
InfoSleuth
407 Citations1997Roberto J. Bayardo, M. Nodine +14 more
This experience paper shows how InfoSleuth integrates new technological developments such as agent technology, domain ontologies, brokerage, and internet computing, in support of mediated interoperation of data and services in a dynamic and open environment.
Integration of heterogeneous databases without common domains using queries based on textual similarity
380 Citations1998William W. Cohen
This paper rejects the assumption that global domains can be easily constructed, and assumes instead that the names are given in natural language text, and proposes a logic called WHIRL which reasons explicitly about the similarity of local names, as measured using the vector-space model commonly adopted in statistical information retrieval.
Infomaster
343 Citations1997Michael Genesereth, Arthur M. Keller +1 more
The core of Infomaster is a facilitator that dynamically determines an efficient way to answer the user's query using as few sources as necessary and harmonizes the heterogeneities among these sources.
Extracting Semistructured Information from the Web.
304 Citations1997J. Hammer, Héctor García-Molina +3 more
A configurable tool for extracting semistructured data from a set of HTML pages and for converting the extracted information into database objects and various ways of improving the functionality of the current prototype are described.
ACM SIGMOD RecordWrapper generation for semi-structured Internet sources
280 Citations1997Naveen Ashish, Craig A. Knoblock
The key idea is to exploit the formatting information in pages from the source to hypothesize the underlying structure of a page and generate a wrapper that facilitates querying of a source and possibly integrating it with other sources.
Modeling Web sources for information integration
172 Citations1998Craig A. Knoblock, Steven Minton +6 more
This work has developed methods for mapping web sources into a simple, uniform representation that makes it efficient to integrate multiple sources and makes it easy to maintain these agents and incorporate new sources as they become available.
ACM SIGMOD RecordThe Araneus Web-based management system
91 Citations1998Giansalvatore Mecca, Paolo Atzeni +3 more
The Araneus Web-based management system
69 Citations1998Giansalvatore Mecca, Paolo Atzeni +3 more
The paper describes the ARANEUS Wel-Base Management System, a system developed at Universitb di Roma Tre, which represents a proposal towards the definition of a new kind of data-repository, designed to manage Web data in the database style.
The distributed information search component (Disco) and the World Wide Web
67 Citations1997Anthony Tomasic, Rémy Amouroux +4 more
The Distributed Information Search COmponent (DISCO) is a prototype heterogeneous distributed database that accesses underlying data sources and implementations of these solutions and operations over a collection of wrappers that access information both in files and on the World Wide Web.
Recognizing structure in Web pages using similarity queries
46 Citations1999William W. Cohen
The structure-recognition methods described can be used to learn a wrapper from examples, or for maintaining a wrapper as a Web page changes format, and in these settings, the top-ranked structure is meaningful nearly 85% of the time.
Initial Results on Wrapping Semistructured Web Pages with Finite-State Transducers and Contextual Rules
45 Citations1998Chun‐Nan Hsu
Experimental results show that the learning algorithm can learn correct wrappers for a wide range of Web pages with a handful of examples and generalize well over unseen pages and structural patterns.
ACM Transactions on Information SystemsCustomizing information capture and access
38 Citations1997Daniela Rus, Devika Subramanian
This article presents a customizable architecture for software agents that capture and access information in large, heterogeneous, distributed electronic repositories to exploit underlying structure at various levels of granularity to build high-level indices with task-specific interpretations.
ACM SIGMOD RecordThe distributed information search component (Disco) and the World Wide Web
14 Citations1997Anthony Tomasic, Rémy Amouroux +4 more
User-oriented smart-cache for the Web
8 Citations1998Zoé Lacroix, Arnaud Sahuguet +1 more
This work proposes a user-oriented approach to querying information on the Web based on the idea that the schema is a posteriori expressed by the user's needs when asking a query, and AKIRA is the only system that uses information retrieval and extraction integrated with database techniques to provide maximum flexibility to the user.