Web spam identification through language model analysis
Published 21 April 2009
Juan Martínez-Romo, Lourdes Araujo
Citations58
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A language model approach is applied to different sources of information extracted from a Web page, in order to provide high quality indicators in the detection of Web Spam and results are a system that improves the detection on two large and public datasets such as WEBSPAM-UK2006 and WEB SPam-UK2007.
Abstract
This paper applies a language model approach to different sources of information extracted from a Web page, in order to provide high quality indicators in the detection of Web Spam. Two pages linked by a hyperlink should be topically related, even though this were a weak contextual relation. For this reason we have analysed different sources of information of a Web page that belongs to the context of a link and we have applied Kullback-Leibler divergence on them for characterising the relationship between two linked pages. Moreover, we combine some of these sources of information in order to obtain richer language models. Given the different nature of internal and external links, in our study we also distinguished these types of links getting a significant improvement in classification tasks. The result is a system that improves the detection of Web Spam on two large and public datasets such as WEBSPAM-UK2006 and WEBSPAM-UK2007.
Keywords
Computer Science
Elements of Information Theory
37,533 Citations2001Thomas M. Cover, Joy A. Thomas
Elsevier eBooksData Mining: Practical Machine Learning Tools and Techniques
25,718 Citations2011Ian H. Witten, Eibe Frank
ACM SIGIR ForumA Language Modeling Approach to Information Retrieval
2,532 Citations2017Jay Ponte, W. Bruce Croft
It will be shown that probabilistic methods can be used to predict topic changes in the context of the task of new event detection and provide further proof of concept for the use of language models for retrieval tasks.
A language modeling approach to information retrieval
1,723 Citations1998Jay Ponte, W. Bruce Croft
This work proposes an approach to retrieval based on probabilistic language modeling and integrates document indexing and document retrieval into a single model, which significantly outperforms standard tf.idf weighting on two different collections and query sets.
Elsevier eBooksCombating Web Spam with TrustRank
1,025 Citations2004Zoltán Gyöngyi, Héctor García-Molina +1 more
This paper proposes techniques to semi-automatically separate reputable, good pages from spam, and shows that they can effectively filter out spam from a significant fraction of the web, based on a good seed set of less than 200 sites.
Detecting spam web pages through content analysis
612 Citations2006Alexandros Ntoulas, Marc Najork +2 more
Some previously-undescribed techniques for automatically detecting spam pages are considered, and the effectiveness of these techniques in isolation and when aggregated using classification algorithms is examined.
No CategoryLucene in action
611 Citations2005Otis Gospodnetić, Erik Hatcher +1 more
Lucene in Action describes what Lucene is and how it works and most importantly how it can be used in a variety of real-world use cases, such at Nutch, an open-source project designed to index the internet very much like Google.
Adversarial Information Retrieval on the WebWeb Spam Taxonomy
488 Citations2005Zoltán Gyöngyi, Héctor García-Molina
This paper presents a comprehensive taxonomy of current spamming techniques, which it is believed can help in developing appropriate countermeasures.
Know your neighbors
313 Citations2007Carlos Castillo, Debora Donato +3 more
A spam detection system that combines link-based and content-based features, and uses the topology of the Web graph by exploiting the link dependencies among the Web pages, which finds that linked hosts tend to belong to the same class.
Spam, damn spam, and statistics
293 Citations2004Dennis Fetterly, Mark S. Manasse +1 more
This paper proposes that some spam web pages can be identified through statistical analysis, and examines a variety of properties, including linkage structure, page content, and page evolution, and finds that outliers in the statistical distribution of these properties are highly likely to be caused by web spam.
Blocking blog spam with language model disagreement
210 Citations2005Gilad Mishne, David Carmel +1 more
An approach for detecting link spam common in blog comments by comparing the language models used in the blog post, the comment, and pages linked by the comments, which requires no training, no hard-coded rule sets, and no knowledge of complete-web connectivity.
ACM SIGIR ForumA reference collection for web spam
205 Citations2006Carlos Castillo, Debora Donato +5 more
This is the first publicly available Web spam collection that includes page contents and links, and that has been labelled by a large and diverse set of judges.
SpamRank -- Fully Automatic Link Spam Detection.
171 Citations2005András A. Benczúr, Károly Csalogány +2 more
Recognizing Nepotistic Links on the Web
150 Citations2000Brian D. Davison
High accuracy in initial experiments is reported to show the potential for using a machine learning tool to automatically recognize and eliminate nepotistic links— links between pages that are present for reasons other than merit.
Link-Based Characterization and Detection of Web Spam
128 Citations2006Luca Becchetti, Carlos Castillo +3 more
Web spam identification through content and hyperlinks
71 Citations2008Jacob Abernethy, Olivier Chapelle +1 more
An algorithm, witch, that learns to detect spam hosts or pages on the Web that simultaneously exploits the structure of the Web graph as well as page contents and features is presented.
Exploring linguistic features for web spam detection
61 Citations2008Jakub Piskorski, Marcin Sydow +1 more
Preliminary analysis seems to indicate that certain linguistic features may be useful for the spam-detection task when combined with features studied elsewhere.
Transductive link spam detection
41 Citations2007Dengyong Zhou, Christopher J. C. Burges +1 more
This work develops discrete analysis on directed graphs, and construct a discrete analogue of classical regularization theory via discrete analysis, and applies this approach to real-world link spam detection problems, and encouraging results have been obtained.
Detecting nepotistic links by language model disagreement
25 Citations2006András A. Benczúr, István Bíró +2 more
The applicability of hyperlink downweighting by means of language model disagreement is demonstrated and various forms of nepotism such as common maintainers, ads, link exchanges or misused affiliate programs are fought.
Measuring similarity to detect qualified links
23 Citations2007Xiaoguang Qi, Lan Nie +1 more
Experiments show this approach almost doubles the performance of Kleinberg's HITS and boosts Bharat and Henzinger's imp algorithm by close to 9% in terms of precision and outperforms a previous approach focusing on link farm detection.