A Corpus-based Probabilistic Grammar with Only Two Non-terminals.

Abstract

The availability of large, syntactically-bracketed corpora such as the Penn Tree Bank affords us the opportunity to automatically build or train broad-coverage grammars, and in particular to train probabilistic grammars. A number of recent parsing experiments have also indicated that grammars whose production probabilities are dependent on the context can be more effective than context-free grammars in selecting a correct parse. To make maximal use of context, we have automatically constructed, from the Penn Tree Bank version 2, a grammar in which the symbols S and NP are the only real nonterminals, and the other non-terminals or grammatical nodes are in effect embedded into the right-hand-sides of the S and NP rules. For example, one of the rules extracted from the tree bank would be S -? NP VBX JJ CC VBX NP [1] (where NP is a non-terminal and the other symbols are terminals -- part-of-speech tags of the Tree Bank). The most common structure in the Tree Bank associated with this expan...