Intent Classification and Slot Filling for Privacy Policies
Published 1 January 2021Open access
Wasi Uddin Ahmad, Jianfeng Chi, Tu Le, Thomas Norton, Yuan Tian, Kai-Wei Chang
Citations24
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work proposes PolicyIE, an English corpus consisting of 5,250 intent and 11,788 slot annotations spanning 31 privacy policies of websites and mobile applications, and presents two alternative neural approaches as baselines, intent classification and slot filling as a joint sequence tagging and modeling them as a sequence-to-sequence (Seq2Seq) learning task.
Abstract
Wasi Ahmad, Jianfeng Chi, Tu Le, Thomas Norton, Yuan Tian, Kai-Wei Chang. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Keywords
Social SciencesDecision SciencesComputer Science
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)MizAR 60 for Mizar 50
76,311 Citations2023Jakubův, Jan, Chvalovský, Karel +7 more
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
Journal of the American Statistical AssociationContent Analysis: An Introduction to its Methodology.
24,566 Citations1984Mack Shelley, Klaus Krippendorff
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case
17,334 Citations2019Yinhan Liu, Myle Ott +8 more
This work considers the task of building machine learning models to automatically select the best combination for a problem instance and contributes to the automatic learning of instance features directly from the high-level representation of a problem instance using a transformer encoder.
ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
12,978 Citations2001John Lafferty, Andrew McCallum +1 more
This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
Transactions of the Association for Computational LinguisticsEnriching Word Vectors with Subword Information
9,784 Citations2017Piotr Bojanowski, Édouard Grave +2 more
A new approach based on the skipgram model, where each word is represented as a bag of character n-grams, with words being represented as the sum of these representations, which achieves state-of-the-art performance on word similarity and analogy tasks.
Transformers: State-of-the-Art Natural Language Processing
8,019 Citations2020Thomas Wolf, Lysandre Debut +20 more
The \textit{Transformers} library is an open-source library that consists of carefully engineered state-of-the art Transformer architectures under a unified API and a curated collection of pretrained models made by and available for the community.
Content Analysis: An Introduction to Its Methodology
5,456 Citations2019Klaus Krippendorff
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
2,589 Citations2016Xuezhe Ma, Eduard Hovy
A novel neutral network architecture is introduced that benefits from both word- and character-level representations automatically, by using combination of bidirectional LSTM, CNN and CRF, thus making it applicable to a wide range of sequence labeling tasks.
Computational LinguisticsInter-Coder Agreement for Computational Linguistics
1,604 Citations2008Ron Artstein, Massimo Poesio
It is argued that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder.
Zenodo (CERN European Organization for Nuclear Research)Transformers: State-of-the-Art Natural Language Processing
1,172 Citations2020Thomas Wolf, Lysandre Debut +14 more
arXiv (Cornell University)Unified language model pre-training for natural language understanding and generation
949 Citations2024Li Dong
A new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks that compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks.
Conference of the European Chapter of the Association for Computational Linguisticsbrat: a Web-based Tool for NLP-Assisted Text Annotation
941 Citations2012Pontus Stenetorp, Sampo Pyysalo +4 more
The brat rapid annotation tool (BRAT) is introduced, an intuitive web-based tool for text annotation supported by Natural Language Processing (NLP) technology and an evaluation of annotation assisted by semantic class disambiguation on a multicategory entity mention annotation task, showing a 15% decrease in total annotation time.
The ATIS spoken language systems pilot corpus
811 Citations1990Charles T. Hemphill, John J. Godfrey +1 more
This pilot marks the first full-scale attempt to collect a corpus to measure progress in Spoken Language Systems that include both a speech and natural language component and provides guidelines for future efforts.
Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling
698 Citations2016Bing Liu, Ian Lane
This work proposes an attention-based neural network model for joint intent detection and slot filling, both of which are critical steps for many speech understanding and dialog systems.
arXiv (Cornell University)MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
629 Citations2020Wenhui Wang, Furu Wei +4 more
This work presents a simple and effective approach to compress large Transformer (Vaswani et al., 2017) based pre-trained models, termed as deep self-attention distillation, and demonstrates that the monolingual model outperforms state-of-the-art baselines in different parameter size of student models.
arXiv (Cornell University)MASS: Masked Sequence to Sequence Pre-training for Language Generation
580 Citations2019Kaitao Song, Xu Tan +3 more
arXiv (Cornell University)Unified Language Model Pre-training for Natural Language Understanding\n and Generation
539 Citations2019Li Dong, Nan Yang +7 more
arXiv (Cornell University)MASS: Masked Sequence to Sequence Pre-training for Language Generation
478 Citations2019Kaitao Song, Xu Tan +3 more
This work proposes MAsked Sequence to Sequence pre-training (MASS) for the encoder-decoder based language generation tasks, which achieves the state-of-the-art accuracy on the unsupervised English-French translation, even beating the early attention-based supervised model.
arXiv (Cornell University)Improving Intent Detection Accuracy Through Token Level Labeling
412 Citations2021Chen Qian, Zhuo Zhu +1 more
This work proposes a joint intent classification and slot filling model based on BERT that achieves significant improvement on intent classification accuracy, slot filling F1, and sentence-level semantic frame accuracy on several public benchmark datasets, compared to the attention-based recurrent neural network models and slot-gated models.
British Journal of General PracticeSolutions to problematic polypharmacy: learning from the expertise of patients
368 Citations2015Joanne Reeve, M. P. Dickenson +7 more
The audience were invited to propose solutions to current concerns about overmedicalisation, treatment burden, and over- and under-diagnosis and two of the final eight proposals related to reducing prescribing.
End-to-end learning of semantic role labeling using recurrent neural networks
366 Citations2015Jie Zhou, Wei Xu
This work proposes to use deep bi-directional recurrent network as an end-to-end system for SRL, which takes only original text information as input feature, without using any syntactic knowledge.
arXiv (Cornell University)Snips Voice Platform: an embedded Spoken Language Understanding system\n for private-by-design voice interfaces
355 Citations2018Alice Coucke, Alaa Saade +10 more
A general framework for information extraction using dynamic span graphs
314 Citations2019Yi Luan, Dave Wadden +4 more
This framework significantly outperforms state-of-the-art on multiple information extraction tasks across multiple datasets reflecting different domains and is good at detecting nested span entities, with significant F1 score improvement on the ACE dataset.
The Creation and Analysis of a Website Privacy Policy Corpus
267 Citations2016Shomir Wilson, Florian Schaub +12 more
A corpus of 115 privacy policies with manual annotations for 23K fine-grained data practices is introduced and the process of using skilled annotators and a purpose-built annotation tool to produce the data is described.
Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog
253 Citations2019Sebastian Schuster, Sonal Gupta +2 more
This paper presents a new data set of 57k annotated utterances in English, Spanish, Spanish and Thai and uses this data set to evaluate three different cross-lingual transfer methods, finding that given several hundred training examples in the the target language, the latter two methods outperform translating the training data.
A joint model of intent determination and slot filling for spoken language understanding
228 Citations2016Xiaodong Zhang, Houfeng Wang
A joint model is proposed based on the idea that the intent and semantic slots of a sentence are correlative, and it outperforms the state-of-the-art approaches on both tasks.
arXiv (Cornell University)UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
225 Citations2020Hangbo Bao, Li Dong +9 more
The experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks across several widely used benchmarks.
arXiv (Cornell University)Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning
174 Citations2018Hamza Harkous, Kassem Fawaz +4 more
Proceedings on Privacy Enhancing TechnologiesMAPS: Scaling Privacy Compliance Analysis to a Million Apps
171 Citations2019Sebastian Zimmeck, Peter Story +6 more
The Mobile App Privacy System (MAPS) is introduced for conducting an extensive privacy census of Android apps and a pipeline for retrieving and analyzing large app populations based on code analysis and machine learning techniques is designed.
(Almost) Zero-Shot Cross-Lingual Spoken Language Understanding
114 Citations2018Shyam Upadhyay, Manaal Faruqui +3 more
Different approaches to train a SLU component with little supervision for two new languages - Hindi and Turkish are examined, and it is shown that with only a few hundred labeled examples the authors can surpass the approaches proposed in the literature.
MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark
111 Citations2021Haoran Li, Abhinav Arora +4 more
A new multilingual dataset, called MTOP, comprising of 100k annotated utterances in 6 languages across 11 domains is presented, and strong zero-shot performance using pre-trained models combined with automatic translation and alignment, and a proposed distant supervision method to reduce the noise in slot label projection are demonstrated.
End-to-End Slot Alignment and Recognition for Cross-Lingual NLU
109 Citations2020Weijia Xu, Batool A Haider +1 more
This work proposes a novel end-to-end model that learns to align and predict slots in a multilingual NLU system and uses the corpus to explore various cross-lingual transfer methods focusing on the zero-shot setting and leveraging MT for language expansion.
Computational LinguisticsReliability Measurement without Limits
108 Citations2008Dennis Reidsma, Jean Carletta
It is demonstrated that the main use of such data, machine learning, can tolerate data with low reliability as long as any disagreement among human coders looks like random noise, which is a commonsense result.
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives
96 Citations2019Abhilasha Ravichander, Alan W. Black +3 more
The PrivacyQA corpus, a corpus consisting of 1750 questions about the privacy policies of mobile applications, and over 3500 expert annotations of relevant answers, offers a challenging corpus for question answering, with genuine real world utility.
The Journal of Legal StudiesAmbiguity in Privacy Policies and the Impact of Regulation
93 Citations2016Joël R. Reidenberg, Jaspreet Bhatia +2 more
This paper compares the impact of different regulatory models on the ambiguity of privacy policies in multiple online sectors and develops a theory of vague and ambiguous terms and a scoring method to compare the relative vagueness of different privacy policies.
Crowdsourcing Annotations for Websites' Privacy Policies
83 Citations2016Shomir Wilson, Florian Schaub +5 more
The results suggest that, if carefully deployed, crowdsourcing can indeed result in the generation of non-trivial annotations and can also help identify elements of ambiguity in policies.
Identifying the Provision of Choices in Privacy Policy Text
81 Citations2017Kanthashree Mysore Sathyendra, Shomir Wilson +3 more
This work presents a two-stage architecture of classification models to identify opt-out choices in privacy policy text, labelling common varieties of choices with a mean F1 score of 0.735, and enables the creation of systems to help Internet users to learn about their choices.
arXiv (Cornell University)Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields
65 Citations2024Ryan Cotterell, Kevin Duh
This paper presents a transfer learning scheme, whereby character-level neural CRFs are trained to predict named entities for both high-resource languages and low- resource languages jointly, improving F1 by up to 9.8 points.
Symposium On Usable Privacy and SecurityHow Short Is Too Short? Implications of Length and Framing on the Effectiveness of Privacy Notices
65 Citations2016Joshua Gluck, Florian Schaub +5 more
The potential of shortening privacy notices by removing privacy practices that a large majority of users are already aware of, and highlighting the implications of described privacy practices with positive or negative framing is examined, suggesting that the length of an effective privacy notice may be bounded.
Finding a Choice in a Haystack: Automatic Extraction of Opt-Out Statements from Privacy Policy Text
64 Citations2020Vinayshekhar Bannihatti Kumar, Roger Iyengar +10 more
The creation of two corpora of opt-out choices are described, which enable the training of classifiers to identify opt-outs in privacy policies and a web browser extension designed to present available opt- out choices to users as they browse the web is introduced.
Unsupervised Alignment of Privacy Policies using Hidden Markov Models
51 Citations2014Rohan Ramanath, Fei Liu +2 more
An unsupervised HMM is applied to the problem of aligning sections of a thousand policy documents, based on the issues they address, and in two new evaluations, the approach is found more effective than clustering and topic models.
Proceedings on Privacy Enhancing TechnologiesAutomated Extraction and Presentation of Data Practices in Privacy Policies
49 Citations2021Duc Hoang Bui, Kang G. Shin +2 more
A fully automated system, called PI-Extract, which accurately extracts privacy practices by a neural model and outperforms, by a large margin, strong rule-based baselines to help users better understand privacy-policy documents is presented.
PolicyQA: A Reading Comprehension Dataset for Privacy Policies
48 Citations2020Wasi Uddin Ahmad, Jianfeng Chi +2 more
This paper argues that providing users with a short text span from policy documents reduces the burden of searching the target information from a lengthy text segment, and evaluates two existing neural QA models and performs rigorous analysis to reveal the advantages and challenges offered by PolicyQA.
Infoscience (Ecole Polytechnique Fédérale de Lausanne)Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning
45 Citations2018Hamza Harkous, Kassem Fawaz +4 more
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)A Step Towards Usable Privacy Policy: Automatic Alignment of Privacy Statements
41 Citations2018Fei Liu, Rohan Ramanath +2 more
A corpus of privacy policies of online websites is considered, and a five-point gap between system and median-human levels of agreement with a consensus annotation is suggested, of which half can be closed with bag of words representations and half requires more sophistication.
Towards an information type lexicon for privacy policies
40 Citations2015Jaspreet Bhatia, Travis D. Breaux
An information type lexicon is constructed from manual, human annotations and an entity extractor based on part-of-speech tagging that has a 31-78% chance of containing a word from any previously seen policy.
Lecture notes in computer scienceWhat Websites Know About You
29 Citations2013Elisa Costante, J Hartog +1 more
This paper proposes a solution which automatically analyzes privacy policy text and shows what personal information is collected, based on the use of Information Extraction techniques and represents a step towards the more ambitious aim of automated grading of privacy policies.
Don’t Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding
26 Citations2020Qile Zhu, Haidar Khan +3 more
A non-autoregressive parser which is based on the insertion transformer to overcome these two issues, which speeds up decoding by 3x while outperforming the autoregressive model and significantly improves cross-lingual transfer in the low-resource setting by 37% compared to autore progressive baseline.
Automated Extraction of Regulated Information Types Using Hyponymy Relations
26 Citations2016Jaspreet Bhatia, Morgan C. Evans +2 more
An approach to automatically construct an information type ontology by identifying information type hyponymy in privacy policies using Tregex patterns, a utility to match regular expressions against constituency parse trees, to automate the extraction of hyponyms from privacy policies.
RECIPE: Applying Open Domain Question Answering to Privacy Policies
12 Citations2018Yan Shvartzshanider, Ananth Balashankar +2 more
It is noticed that combining the results of this model with a neural dependency parser based approach yields a significantly higher F1 score of 92.35 compared to manual annotations, which indicates that future work which in-corporates signals from parsing like NLP tasks more explicitly can generalize better on out-of-domain tasks.
Identifying and Classifying Third-party Entities in Natural Language Privacy Policies
10 Citations2020Mitra Bokaie Hosseini, K C Pragyan +2 more
This paper aims to develop an automated approach to extract and categorize third party data recipients (i.e., entities) declared in privacy policies, and analyzes 100 privacy policies associated with most downloaded apps in the Google Play Store.
International Conference on Computational LinguisticsSemantic Parsing for Technical Support Questions
7 Citations2018Abhirut Gupta, Anupama Ray +4 more
An approach for semantic parsing of technical questions that uses grammatical structure to extract these attributes as a baseline, and a CRF based model that can improve performance considerably in the presence of annotated data for training are presented.