Experiments on logistic regression
Stochastic Gradient Descent and 2-Norm Regularization techniques are implemented and a new trick of modifying Sigmoid transfer function in the training stage is described and implemented, which is shown could help reducing overfitting as well.
Abstract
In this report, several experiments have been conducted on a spam data set with Logistic Regression based on Gradient Descent approach. First, the overfitting effect is shown with basic settings (vanilla version). Then Stochastic Gradient Descent and 2-Norm Regularization techniques are both implemented with demonstration of the benefits of these two methods in preventing overfitting. Besides, a new trick of modifying Sigmoid transfer function in the training stage is described and implemented, which is shown could help reducing overfitting as well. At last, the Label Shrinking technique is implemented. Yet, the results on this data set are not satisfying.