Recursive automatic algorithm selection for inductive learning

An approach to overcoming this problem by applying knowledge about the biases of a set of learning algorithms to conduct a recursive automatic algorithm search, which permits classifiers learned by the available algorithms to be mixed in a recursive tree-structured hybrid, thereby allowing different subproblems of the learning task to be learned by different algorithms.

Abstract

The results of empirical comparisons of existing learning algorithms illustrate that each algorithm has a selective superiority; each is best for some but not all tasks. Selective superiority arises because each learning algorithm searches within a restricted generalization space, defined by its representation language, and employs a search bias for selecting a generalization in that space. Given a data set, it is often not clear beforehand which algorithm will yield the best performance. The problem is complicated further because for some learning tasks, different subtasks are learned best using different algorithms. In such cases, the ability to form a hybrid classifier that combines different representation languages will produce a more accurate classifier than employing a single representation language and search bias. This dissertation presents an approach to overcoming this problem by applying knowledge about the biases of a set of learning algorithms to conduct a recursive automatic algorithm search. The approach permits classifiers learned by the available algorithms to be mixed in a recursive tree-structured hybrid, thereby allowing different subproblems of the learning task to be learned by different algorithms. The Model Class Selection System (MCS), an implementation of the approach, combines decision trees, linear discriminant functions and instance-based classifiers in a tree-structured hybrid classifier. Heuristic knowledge about the characteristics that indicate one bias is better than another is encoded in the rule base that guides MCS's search for the best classifier. An empirical evaluation illustrates that MCS achieves classification accuracies equal to or higher than the best of its primitive learning components for a variety of data sets, demonstrating that domain-independent knowledge about the biases of machine learning algorithms can guide an automatic algorithm selection search.