The Modified Nystrom Method: Theories, Algorithms, and Extension.

Tests on real-world datasets demonstrate that the proposed column selection algorithm is both efficient and accurate and that the SS-Nystr\"om method always leads to much higher kernel approximation accuracy than the standard/modified Nystr \"om method.

Abstract

Many kernel methods suffer from high time and space complexities, so they are prohibitive in big-data applications. To tackle the computational challenge, the Nyström method has been extensively used to reduce time and space complexities by sacrificing some accuracy. The Nyström method speedups computation by constructing an approximation of the kernel matrix in question using only a few columns of the matrix. Recently, a variant of the Nyström method called the modified Nyström method has demonstrated significant improvement over the standard Nyström method in approximation accuracy, both theoretically and empirically. In this paper we provide theoretical analysis, efficient algorithms, and a simple but highly accurate extension for the modified Nyström method. First, we prove that the modified Nyström method is exact under certain conditions, and we establish a lower error bound for the modified Nyström method. Second, we develop two efficient algorithms to make the modified Nyström method efficient and practical. We devise a simple column selection algorithm with a provable error bound. With the selected columns at hand, we propose an algorithm that computes the modified Nyström approximation in lower time complexity than the approach in the previous work. Third, the extension which we call the SS-Nyström method has much stronger error bound than the modified Nyström method, especially when the spectrum of the kernel matrix decays slowly. Our proposed SS-Nyström can be computed nearly as efficiently as the modified Nyström method. Finally, experiments on real-world datasets demonstrate that the proposed column selection algorithm is both efficient and accurate and that the SS-Nyström method always leads to much higher kernel approximation accuracy than the standard/modified Nyström method.