An Experimental Evaluation of Spam Filter Performance and Robustness Against Attack
Published 3 August 2006
S. Webb, S. Chitti, Calton Pu
Citations26
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper shows experimentally that learning filters are able to classify large corpora of spam and legitimate email messages with a high degree of accuracy, and introduces an effective attack against these filters which successfully degrades their ability to classify spam.
Abstract
In this paper, we show experimentally that learning filters are able to classify large corpora of spam and legitimate email messages with a high degree of accuracy. The corpora in our experiments contain about half a million spam messages and a similar number of legitimate messages, making them two orders of magnitude larger than the corpora used in current research. The use of such large corpora represents a collaborative approach to spam filtering because the corpora combine spam and legitimate messages from many different sources. First, we show that this collaborative approach creates very accurate spam filters. Then, we introduce an effective attack against these filters which successfully degrades their ability to classify spam. Finally, we present an effective solution to the above attack which involves retraining the filters to accurately identify the attack messages
Keywords
Computer Science
The Nature of Statistical Learning Theory
39,279 Citations1995Vladimir Vapnik
Lecture notes in computer scienceText categorization with Support Vector Machines: Learning with many relevant features
7,925 Citations1998Thorsten Joachims
SVMs achieve substantial improvements over the currently best performing methods and behave robustly over a variety of di-erent learning tasks, eliminating the need for manual parameter tuning.
The Annals of StatisticsAdditive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)
6,842 Citations2000Jerome H. Friedman, Trevor Hastie +1 more
This work shows that this seemingly mysterious phenomenon of boosting can be understood in terms of well-known statistical principles, namely additive modeling and maximum likelihood, and develops more direct approximations and shows that they exhibit nearly identical results to boosting.
A Comparative Study on Feature Selection in Text Categorization
4,766 Citations1997Yiming Yang, Jan Pedersen
DF thresholding, the simplest method with the lowest cost in computation, can be reliably used instead of IG or CHI when the computation of these measures are too expensive, and strong correlations between the DF, IG and CHI values of a term are found.
A comparison of event models for naive bayes text classification
3,224 Citations1998Andrew McCallum, Kamal Nigam
It is found that the multi-variate Bernoulli performs well with small vocabulary sizes, but that the multinomial performs usually performs even better at larger vocabulary sizes--providing on average a 27% reduction in error over the multi -variateBernoulli model at any vocabulary size.
Inductive learning algorithms and representations for text categorization
1,465 Citations1998Susan Dumais, John Platt +2 more
A comparison of the effectiveness of five different automatic learning algorithms for text categorization in terms of learning speed, realtime classification speed, and classification accuracy is compared.
IEEE Transactions on Neural NetworksSupport vector machines for spam categorization
1,457 Citations1999Harris Drucker, Donghui Wu +1 more
The use of support vector machines in classifying e-mail as spam or nonspam is studied by comparing it to three other classification algorithms: Ripper, Rocchio, and boosting decision trees, which found SVM's performed best when using binary features.
A Bayesian Approach to Filtering Junk E-Mail
1,153 Citations1998Mehran Sahami, Susan Dumais +2 more
This work examines methods for the automated construction of filters to eliminate such unwanted messages from a user’s mail stream, and shows the efficacy of such filters in a real world usage scenario, arguing that this technology is mature enough for deployment.
Adversarial classification
893 Citations2004Nilesh Dalvi, Pedro Domingos +3 more
This paper views classification as a game between the classifier and the adversary, and produces a classifier that is optimal given the adversary's optimal strategy, and experiments show that this approach can greatly outperform a classifiers learned in the standard way.
ACM Transactions on Information SystemsAutomated learning of decision rules for text categorization
867 Citations1994Chidanand Apté, Fred J. Damerau +1 more
It is shown that machine-generated decision rules appear comparable to human performance, while using the identical rule-based representation, and compared with other machine-learning techniques.
Conference on Email and Anti-SpamIntroducing the Enron Corpus.
542 Citations2004Bryan Klimt, Yiming Yang
The goal in this paper is to analyze the suitability of this corpus for exploring how to classify messages as organized by a human, so these folders would have likely been misleading.
arXiv (Cornell University)An evaluation of Naive Bayesian anti-spam filtering
527 Citations2000Ion Androutsopoulos, John Koutsias +3 more
It is reached that additional safety nets are needed for the Naive Bayesian anti-spam filter to be viable in practice.
arXiv (Cornell University)Boosting Trees for Anti-Spam Email Filtering
342 Citations2001Xavier Carreras, Lluı́s Màrquez
The boosting-based methods clearly outperform the baseline learning algorithms on the PU1 corpus, achieving very high levels of the F1 measure and obtaining better ``high-precision'' classifiers, which is a very important issue when misclassification costs are considered.
ACM Transactions on Asian Language Information ProcessingAn evaluation of statistical spam filtering techniques
317 Citations2004Le Zhang, Jingbo Zhu +1 more
Experiments show that classifiers using features from message header alone can achieve comparable or better performance than filters utilizing body features only, which implies that message headers can be reliable and powerfully discriminative feature sources for spam filtering.
A comparison of event models for Naive Bayes anti-spam e-mail filtering
208 Citations2003Karl-Michael Schneider
It is found that the multinomial model is less biased towards one class and achieves slightly higher accuracy than the multi-variate Bernoulli model.
On Attacking Statistical Spam Filters.
173 Citations2004Gregory L. Wittel, Shyhtsun Felix Wu
The general attack methods spammers use are examined, along with challenges faced by developers and spammers, and an attack is demonstrated that attempts to more strongly work against the statistical nature behind spam systems.
Learning to Filter Unsolicited Commercial E-Mail
133 Citations2006Ion Androutsopoulos, Γεώργιος Παλιούρας +2 more
National Conference on Artificial IntelligenceSpamCop: A Spam Classification & Organisation Program
85 Citations1998Patrick Pantel, Dekang Lin
This work presents a simple, yet highly accurate, spam filtering program, called SpamCop, which is able to identify about 92% of the spams while misclassifying only about 1.16%" of the nonspam e-mails.
An Experimental Evaluation of Spam Filter Performance and Robustness Against Attack
26 Citations2006S. Webb, S. Chitti +1 more
This paper shows experimentally that learning filters are able to classify large corpora of spam and legitimate email messages with a high degree of accuracy, and introduces an effective attack against these filters which successfully degrades their ability to classify spam.