Characterizing Web Spam Using Content and HTTP Session Analysis.

Conference on Email and Anti-SpamPublished 1 January 2007

Steve Webb, James Caverlee, Calton Pu

Citations25

TL;DR

The first large-scale characterization of web spam using content and HTTP session analysis techniques on the Webb Spam Corpus is performed, showing significant concentration of hosting IP addresses in two narrow ranges as well as significant overlaps among session header values.

Abstract

Web spam research has been hampered by a lack of statistically significant collections. In this paper, we perform the first large-scale characterization of web spam using content and HTTP session analysis techniques on the Webb Spam Corpus ‐ a collection of about 350,000 web spam pages. Our content analysis results are consistent with the hypothesis that web spam pages are dierent from normal web pages, showing far more duplication of physical content and URL redirections. An analysis of session information collected during the crawling of the Webb Spam Corpus shows significant concentration of hosting IP addresses in two narrow ranges as well as significant overlaps among session header values. These findings suggest that content and HTTP session analysis may contribute a great deal towards future eorts to automatically distinguish web spam pages from normal web pages.

Keywords

Computer Science

Computer Networks and ISDN SystemsSyntactic clustering of the Web

1,347 Citations1997Andrei Broder, S. Glassman +2 more

An efficient way to determine the syntactic similarity of files is developed and applied to every document on the World Wide Web, and a clustering of all the documents that are syntactically similar is built.

Detecting spam web pages through content analysis

612 Citations2006Alexandros Ntoulas, Marc Najork +2 more

Some previously-undescribed techniques for automatically detecting spam pages are considered, and the effectiveness of these techniques in isolation and when aggregated using classification algorithms is examined.

Adversarial Information Retrieval on the WebWeb Spam Taxonomy

488 Citations2005Zoltán Gyöngyi, Héctor García-Molina

This paper presents a comprehensive taxonomy of current spamming techniques, which it is believed can help in developing appropriate countermeasures.

Spam, damn spam, and statistics

293 Citations2004Dennis Fetterly, Mark S. Manasse +1 more

This paper proposes that some spam web pages can be identified through statistical analysis, and examines a variety of properties, including linkage structure, page content, and page evolution, and finds that outliers in the statistical distribution of these properties are highly likely to be caused by web spam.

ACM SIGIR ForumA reference collection for web spam

205 Citations2006Carlos Castillo, Debora Donato +5 more

This is the first publicly available Web spam collection that includes page contents and links, and that has been labelled by a large and diverse set of judges.

Software Practice and ExperienceA large‐scale study of the evolution of Web pages

202 Citations2004Dennis Fetterly, Mark S. Manasse +2 more

View PDF

Detecting phrase-level duplication on the world wide web

127 Citations2005Dennis Fetterly, Mark S. Manasse +1 more

The algorithms used to discover a number of other instances of large-scale phrase-level replication within the two data sets collected in December 2002 and June 2004 are described.

Cloaking and Redirection: A Preliminary Study.

104 Citations2005Baoning Wu, Brian D. Davison

This work downloaded two sets of Web pages while mimicking a popular Web crawler and as a common Web browser to understand cloaking and redirection on the Web and proposes a method of detecting cloaking pages by calculating the di(cid:11)erence of three copies of the same page.

Introducing the Webb Spam Corpus: Using Email Spam to Identify Web Spam Automatically

98 Citations2006Steve Webb, James Caverlee +1 more

Since the Webb Spam Corpus bridges the worlds of email spam and Web spam, it is noted that it can be used to aid traditional email spam classication algorithms through an analysis of the characteristics of the Web pages referenced by email messages.

Spam double-funnel

89 Citations2007Yi‐Min Wang, Ming Ma +2 more

A five-layer, double-funnel model for describing end-to-end redirection spam is proposed, a methodology for analyzing the layers is presented, and prominent domains on each layer are identified using two sets of commercial keywords.