A preprocessing method of internet search data for prediction improvement

A comprehensive method for Internet search data preprocessing, which includes the critical steps: keywords selection, time difference measurement, and leading index composition, is developed and can get the leading keywords index with stable leading relation and high degree of fit for Chinese stock market price.

Abstract

The correlations between Internet search data and socio-economic Indicators have been proved in many studies, but the basis work of these studies - data preprocessing, determining the quality of the result, has lacked a systematic methodology. In this paper, we develop a comprehensive method for Internet search data preprocessing, which includes the critical steps: (a) keywords selection, (b) time difference measurement, and (c) leading index composition. Applying our method to study Chinese stock market price, we can get the leading keywords index with stable leading relation and high degree of fit. Specifically, the correlation coefficient between our leading keywords index and Shanghai Composite Index reaches 98.7%, and Granger test confirms that keywords index has significant prediction ability for Shanghai Composite Index. Adding keywords index to the AR model can reduce the MAPE from 3.8% to 1.4%, and each percentage point change of keywords index is correlated with 0.136 percentage point move in the same direction of Shanghai Composite Index in next period.