Automatic segmentation of text into structured records
Published 1 May 2001
Vinayak Borkar, Kaustubh Deshmukh, Sunita Sarawagi
Citations212
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A tool DATAMOLD is described that learns to automatically extract structure when seeded with a small number of training examples and enhances on Hidden Markov Models (HMM) to build a powerful probabilistic model that corroborates multiple sources of information.
Abstract
In this paper we present a method for automatically segmenting unformatted text records into structured elements. Several useful data sources today are human-generated as continuous text whereas convenient usage requires the data to be organized as structured records. A prime motivation is the warehouse address cleaning problem of transforming dirty addresses stored in large corporate databases as a single text field into subfields like "City" and "Street". Existing tools rely on hand-tuned, domain-specific rule-based systems.
Keywords
Computer ScienceDecision Sciences
Proceedings of the IEEEA tutorial on hidden Markov models and selected applications in speech recognition
22,785 Citations1989L. R. Rabiner
Fundamentals of speech recognition
7,687 Citations1993Lawrence R. Rabiner, Biing‐Hwang Juang
This book presents a meta-modelling framework for speech recognition that automates the very labor-intensive and therefore time-heavy and therefore expensive and expensive process of manually modeling speech.
Maximum Entropy Markov Models for Information Extraction and Segmentation
1,333 Citations2000Andrew McCallum, Dayne Freitag +1 more
A new Markovian sequence model is presented that allows observations to be represented as arbitrary overlapping features (such as word, capitalization, formatting, part-of-speech), and defines the conditional probability of state sequences given observation sequences.
Wrapper induction for information extraction
1,044 Citations1997Nicholas Kushmerick, Daniel S. Weld
This work introduces wrapper induction, a method for automatically constructing wrappers, and identifies hlrt, a wrapper class that is e(cid:14)ciently learnable, yet expressive enough to handle 48% of a recently surveyed sample of Internet resources.
Machine LearningLearning Information Extraction Rules for Semi-Structured and Free Text
928 Citations1999Stephen Soderland
WHISK is designed to handle text styles ranging from highly structured to free text, including text that is neither rigidly formatted nor composed of grammatical sentences, and can also handle extraction from free text such as news stories.
The merge/purge problem for large databases
804 Citations1995Mauricio A. Hernández, Salvatore J. Stolfo
This paper details the sorted neighborhood method that is used by some to solve merge/purge and presents experimental results that demonstrates this approach may work well in practice but at great expense, and shows a means of improving the accuracy of the results based upon a multi-pass approach.
ComputerDigital libraries and autonomous citation indexing
614 Citations1999Sandra Lawrence, C. Lee Giles +1 more
Digital libraries incorporating ACI can help organize scientific literature and may significantly improve the efficiency of dissemination and feedback and speed the transition to scholarly electronic publishing.
Relational Learning of Pattern-Match Rules for Information Extraction.
494 Citations1997Mary Elaine Califf, Raymond J. Mooney
The field matching problem: Algorithms and applications
482 Citations1996Alvaro Monge, Charles Elkan
Three field matching algorithms are described, one of which is the well-known Smith-Waterman algorithm for comparing DNA and protein sequences, and their performance on real-world datasets is evaluated.
XWRAP: an XML-enabled wrapper construction system for Web information sources
427 Citations2002Liu Liu, Calton Pu +1 more
The paper describes the methodology and the software development of XWRAP, an XML-enabled wrapper construction system for semi-automatic generation of wrapper programs, and introduces and develops a two-phase code generation framework.
Computer Speech & LanguageRobust part-of-speech tagging using a hidden Markov model
416 Citations1992Julian Kupiec
A system for part-of-speech tagging is described, based on a hidden Markov model which can be trained using a corpus of untagged text, which results in a model that correctly tags approximately 96% of the text.
Information SystemsGenerating finite-state transducers for semi-structured data extraction from the Web
414 Citations1998Chun‐Nan Hsu, Ming-Tzung Dung
This paper presents SoftMealy, a novel wrapper representation formalism based on a finite-state transducer and contextual rules that can wrap a wide range of semistructured Web pages because FSTs can encode each different attribute permutation as a path.
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)Learning Hidden Markov Model Structure for Information Extraction
384 Citations2018Kristie Seymore, Roni Rosenfeld
It is demonstrated that a manually-constructed model that contains multiple states per extraction field outperforms a model with one state per field, and the use of distantly-labeled data to set model parameters provides a significant improvement in extraction accuracy.
A hierarchical approach to wrapper induction
376 Citations1999Ion Muslea, Steve Minton +1 more
This work introduces an inductive algorithm, STALKER, that generates high accuracy extraction rules based on user-labeled training examples that can handle information sources that could not be wrapped by existing techniques.
NoDoSE—a tool for semi-automatically extracting structured and semistructured data from text documents
303 Citations1998Brad Adelberg
NoDoSE, the Northwestern Document Structure Extractor, is described, which is an interactive tool for semi-automatically determining the structure of such documents and then extracting their data.
Information Extraction with HMM Structures Learned by Stochastic Optimization
286 Citations2000Dayne Freitag, Andrew McCallum
This paper demonstrates that extraction accuracy strongly depends on the selection of structure, and presents an algorithm for automatically finding good structures by stochastic optimization, which finds HMM models that almost always out-perform a fixed model, and have superior average performance across tasks.
The Intelligent Enterprise
237 Citations2014
ACM SIGMOD RecordRecord-boundary discovery in Web documents
234 Citations1999David W. Embley, Yi Jiang +1 more
Extraction Patterns for Information Extraction Tasks: A Survey
227 Citations1999Ion Muslea
This paper surveys the various types of extraction patterns that are generated by machine learning algorithms and identifies three main categories of patterns, which cover a variety of application domains, and compares and contrast the patterns from each category.
Very Large Data BasesBuilding Light-Weight Wrappers for Legacy Web Data-Sources Using W4F
167 Citations1999Arnaud Sahuguet, Fabien Azavant
The Web has become a major conduit to information repositories of all kinds, but Web data sources also consist of standalone HTML pages hand-coded by individuals, that provide very useful information such as reviews, digests, links, etc.
Lecture notes in computer scienceLearning information extraction patterns from examples
146 Citations1996Scott B. Huffman
A system that can learn dictionaries of extraction patterns directly from user-provided examples of texts and events to be extracted from them, and learns patterns that recognize relationships between key constituents based on local syntax.
Record-boundary discovery in Web documents
137 Citations1999David W. Embley, Yi Jiang +1 more
This paper describes a heuristic approach to discovering record boundaries in Web documents that captures the structure of a document as a tree of nested HTML tags, and locates the subtree containing the records of interest using five independent heuristics and selects a consensus separator tag based on a combined heuristic.
Araneus in the Era of XML
59 Citations1999Giansalvatore Mecca, Paolo Merialdo +1 more
The impact of XML on the research in the last few years by this group in the framework of the A RANEUS project is discussed, including how XML fits in this framework, how it is influencing the authors' ideas and their way of thinking about Web data, and the design choices needed to adapt their tools – originally conceived for HTML – to the management of XML data.
RePEc: Research Papers in EconomicsLearning information extraction patterns from examples
22 Citations1995Scott B. Huffman
TheaterLoc: using information integration technology to rapidly build virtual applications
14 Citations2002Greg Barish, Yi-Shin Chen +5 more
TheaterLoc is demonstrated, an information integration application that allows users to retrieve information about theatres and restaurants for various U.S. cities, including an interactive map depicting their relative locations, from five distinct heterogeneous and distributed sources.