Do Characters Abuse More Than Words?
Published 1 January 2016Open access
Yashar Mehdad, Joel Tetreault
Citations195
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This study investigates the effectiveness of word and character n-grams for abusive language detection in user-generated online comments and shows the power of character-based features for detecting abusive language.
Abstract
Although word and character n-grams have been used as features in different NLP applications, no systematic comparison or analysis has shown the power of character-based features for detecting abusive language.In this study, we investigate the effectiveness of such features for abusive language detection in user-generated online comments, and show that such methods outperform previous state-of-theart approaches and other strong baselines.
Keywords
Computer ScienceSocial Sciences
arXiv (Cornell University)Distributed Representations of Words and Phrases and their Compositionality
18,086 Citations2013Tomáš Mikolov, Ilya Sutskever +3 more
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
18,050 Citations2013Tomáš Mikolov, Kai Chen +2 more
Two novel model architectures for computing continuous vector representations of words from very large data sets are proposed and it is shown that these vectors provide state-of-the-art performance on the authors' test set for measuring syntactic and semantic word similarities.
Recurrent neural network based language model
5,349 Citations2010Tomáš Mikolov, Martin Karafiát +3 more
Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model.
arXiv (Cornell University)Distributed Representations of Sentences and Documents
5,128 Citations2014Quoc V. Le, Tomáš Mikolov
Paragraph Vector is an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents, and its construction gives the algorithm the potential to overcome the weaknesses of bag-of-words models.
arXiv (Cornell University)Embedding Entities and Relations for Learning and Inference in Knowledge Bases
2,051 Citations2014d'Amato, Claudia, Wen-tau Yih +2 more
It is found that embeddings learned from the bilinear objective are particularly good at capturing relational semantics and that the composition of relations is characterized by matrix multiplication.
Abusive Language Detection in Online User Content
1,107 Citations2016Chikashi Nobata, Joel Tetreault +3 more
A machine learning based method to detect hate speech on online user comments from two domains which outperforms a state-of-the-art deep learning approach and a corpus of user comments annotated for abusive language, the first of its kind.
Baselines and Bigrams: Simple, Good Sentiment and Topic Classification
967 Citations2012Sida Wang, Christopher D. Manning
It is shown that the inclusion of word bigram features gives consistent gains on sentiment analysis tasks, and a simple but novel SVM variant using NB log-count ratios as feature values consistently performs well across tasks and datasets.
Hate Speech Detection with Comment Embeddings
712 Citations2015Nemanja Djuric, Jing Zhou +4 more
This work proposes to learn distributed low-dimensional representations of comments using recently proposed neural language models, that can then be fed as inputs to a classification algorithm, resulting in highly efficient and effective hate speech detectors.
Detecting Offensive Language in Social Media to Protect Adolescent Online Safety
630 Citations2012Ying Chen, Yilu Zhou +2 more
This work proposes the Lexical Syntactic Feature (LSF) architecture to detect offensive content and identify potential offensive users in social media, and incorporates a user's writing style, structure and specific cyber bullying content as features to predict the user's potentiality to send out offensive content.
Detecting Hate Speech on the World Wide Web
514 Citations2012William B. Warner, Julia Hirschberg
The definition of hate speech, the collection and annotation of the hate speech corpus, and a mechanism for detecting some commonly used methods of evading common "dirty word" filters are described.
Lecture notes in computer scienceImproving Cyberbullying Detection with User Context
382 Citations2013Maral Dadvar, Dolf Trieschnigg +2 more
It is shown that taking user context into account improves the detection of cyberbullying.
Detection of Harassment on Web 2.0
335 Citations2009Dawei Yin, Zhen‐Zhen Xue +3 more
This paper uses a supervised learning approach for harassment that employs content features, sentiment features, and contextual features of documents and achieves significant improvements over several baselines, including Term Frequency- Inverse Document Frequency (TFIDF) approaches.
Detecting offensive tweets via topical feature discovery over a large scale twitter corpus
270 Citations2012Guang Xiang, Bin Fan +3 more
This work proposes a novel semi-supervised approach for detecting profanity-related offensive content in Twitter that exploits linguistic regularities in profane language via statistical topic modeling on a huge Twitter corpus, and detects offensive tweets using automatically these generated features.
IEEE Automatic Speech Recognition and Understanding WorkshopRNNLM - Recurrent Neural Network Language Modeling Toolkit
262 Citations2011Tomáš Mikolov, Stefan Kombrink +3 more
A freely available open-source toolkit for training recurrent neural network based language models that can be easily used to improve existing speech recognition and machine translation systems and used as a baseline for future research of advanced language modeling techniques.
A Report on the 2017 Native Language Identification Shared Task
158 Citations2017Shervin Malmasi, Keelan Evanini +6 more
The fusion track showed that combining the written and spoken responses provides a large boost in prediction accuracy, and multiple classifier systems were the most effective in all tasks, with most based on traditional classifiers with lexical/syntactic features.
Not All Character N-grams Are Created Equal: A Study in Authorship Attribution
156 Citations2015Upendra Sapkota, Steven Bethard +2 more
It is demonstrated that characterngrams that capture information about affixes and punctuation account for almost all of the power of character n-grams as features.
arXiv (Cornell University)Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
95 Citations2014Grégoire Mesnil, Tomáš Mikolov +2 more
This work shows how to use for this task the standard generative models, which are slightly complementary to the state of the art techniques, and achieves strong results on a well-known dataset of IMDB movie reviews.
Combining Word-Level and Character-Level Models for Machine Translation Between Closely-Related Languages
87 Citations2012Preslav Nakov, Jörg Tiedemann
This work uses character-level translation trained on n-gram-character-aligned bitexts and tuned using word-level BLEU to augment with character-based transliteration at the word level and combine with a word- level translation model.
Hierarchical Neural Language Models for Joint Representation of Streaming Documents and their Content
75 Citations2015Nemanja Djuric, Hao Wu +3 more
Extensions to the proposed model, which can be applied to personalized recommendation and social relationship mining by adding further user layers to the hierarchy, thus learning user-specific vectors to represent individual preferences, are discussed.
National Conference on Artificial IntelligenceUsing Crowdsourcing to Improve Profanity Detection
71 Citations2012Sara Owsley Sood, Judd Antin +1 more
This article presents work that attempts to move beyond list-based profanity detection systems by identifying the context in which profanity occurs, and uses a set of comments from a social news site labeled by Amazon Mechanical Turk workers for the presence of profanity.
Fast Matrix-Vector Multiplications for Large-Scale Logistic Regression on Shared-Memory Systems
27 Citations2015Mu-Chu Lee, Wei-Lin Chiang +1 more
This work investigates the use of parallel sparse matrix-vector multiplications in a Newton method for large scale logistic regression and indicates that under suitable settings excellent speedup can be achieved.
Lecture notes in computer scienceWhat We Can Learn from Looking at Profanity
12 Citations2014Gustavo Laboreiro, Eugénio Oliveira
A large number of swearing words are found in English tweets and Yahoo! Buzz messages, and efforts in recognizing, understanding and dealing with profanity do not share resources, namely, their dataset, which imposes duplication of effort and non-comparable results.