PLIS: Proposed Language Independent Stemmer for Information Retrieval Systems Using Dynamic Programming

This paper proposes an unsupervised stemming which is hybridized with partial lemmatization for four morphologically different languages such as English, French, Tamil and Hindi.

Abstract

Stemming and Lemmatization are two significant natural language processing techniques extensively used in Information Retrieval for query processing and Machine Translation for reducing the data sparseness. Most of the existing Stemmers and Lemmatizers are based on some language dependent rules which require the supervision of a language expert. Some probabilistic approach needs vast amounts of monolingual corpus. Both Stemming and Lemmatization minimize inflectional structure, and sometimes derivationally related structure of a word to a common base form. This paper proposes an unsupervised stemming which is hybridized with partial lemmatization for four morphologically different languages such as English, French, Tamil and Hindi. An innovative attempt is being made to develop a stemming algorithm for a novel conflation method that exploits the quality of words and uses some standard Natural Language Processing tools like Levenshtein Distance and Longest Common Subsequence for Stemming process. This approach can even support other Indian and Non-Indian languages.