PolicyQA: A Reading Comprehension Dataset for Privacy Policies
Published 1 January 2020Open access
Wasi Uddin Ahmad, Jianfeng Chi, Yuan Tian, Kai-Wei Chang
Citations48
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper argues that providing users with a short text span from policy documents reduces the burden of searching the target information from a lengthy text segment, and evaluates two existing neural QA models and performs rigorous analysis to reveal the advantages and challenges offered by PolicyQA.
Abstract
Privacy policy documents are long and verbose. A question answering (QA) system can assist users in finding the information that is relevant and important to them. Prior studies in this domain frame the QA task as retrieving the most relevant text segment or a list of sentences from the policy document given a question. On the contrary, we argue that providing users with a short text span from policy documents reduces the burden of searching the target information from a lengthy text segment. In this paper, we present PolicyQA, a dataset that contains 25,017 reading comprehension style examples curated from an existing corpus of 115 website privacy policies. PolicyQA provides 714 human-annotated questions written for a wide range of privacy practices. We evaluate two existing neural QA models and perform rigorous analysis to reveal the advantages and challenges offered by PolicyQA.
Keywords
Computer ScienceDecision SciencesSocial Sciences
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at Scale
45,681 Citations2018Kefallinos, Dionysios, Alexandris, Georgios +6 more
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
Transactions of the Association for Computational LinguisticsEnriching Word Vectors with Subword Information
9,784 Citations2017Piotr Bojanowski, Édouard Grave +2 more
A new approach based on the skipgram model, where each word is represented as a bag of character n-grams, with words being represented as the sum of these representations, which achieves state-of-the-art performance on word similarity and analogy tasks.
SQuAD: 100,000+ Questions for Machine Comprehension of Text
6,328 Citations2016Pranav Rajpurkar, Jian Zhang +2 more
A strong logistic regression model is built, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%).
arXiv (Cornell University)Bidirectional Attention Flow for Machine Comprehension
1,290 Citations2016Min Joon Seo, Aniruddha Kembhavi +2 more
British Journal of General PracticeSolutions to problematic polypharmacy: learning from the expertise of patients
368 Citations2015Joanne Reeve, M. P. Dickenson +7 more
The audience were invited to propose solutions to current concerns about overmedicalisation, treatment burden, and over- and under-diagnosis and two of the final eight proposals related to reducing prescribing.
The Creation and Analysis of a Website Privacy Policy Corpus
267 Citations2016Shomir Wilson, Florian Schaub +12 more
A corpus of 115 privacy policies with manual annotations for 23K fine-grained data practices is introduced and the process of using skilled annotators and a purpose-built annotation tool to produce the data is described.
arXiv (Cornell University)Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning
174 Citations2018Hamza Harkous, Kassem Fawaz +4 more
Proceedings on Privacy Enhancing TechnologiesMAPS: Scaling Privacy Compliance Analysis to a Million Apps
171 Citations2019Sebastian Zimmeck, Peter Story +6 more
The Mobile App Privacy System (MAPS) is introduced for conducting an extensive privacy census of Android apps and a pipeline for retrieving and analyzing large app populations based on code analysis and machine learning techniques is designed.
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives
96 Citations2019Abhilasha Ravichander, Alan W. Black +3 more
The PrivacyQA corpus, a corpus consisting of 1750 questions about the privacy policies of mobile applications, and over 3500 expert annotations of relevant answers, offers a challenging corpus for question answering, with genuine real world utility.
The Journal of Legal StudiesAmbiguity in Privacy Policies and the Impact of Regulation
93 Citations2016Joël R. Reidenberg, Jaspreet Bhatia +2 more
This paper compares the impact of different regulatory models on the ambiguity of privacy policies in multiple online sectors and develops a theory of vague and ambiguous terms and a scoring method to compare the relative vagueness of different privacy policies.
Crowdsourcing Annotations for Websites' Privacy Policies
83 Citations2016Shomir Wilson, Florian Schaub +5 more
The results suggest that, if carefully deployed, crowdsourcing can indeed result in the generation of non-trivial annotations and can also help identify elements of ambiguity in policies.
Identifying the Provision of Choices in Privacy Policy Text
81 Citations2017Kanthashree Mysore Sathyendra, Shomir Wilson +3 more
This work presents a two-stage architecture of classification models to identify opt-out choices in privacy policy text, labelling common varieties of choices with a mean F1 score of 0.735, and enables the creation of systems to help Internet users to learn about their choices.
Unsupervised Alignment of Privacy Policies using Hidden Markov Models
51 Citations2014Rohan Ramanath, Fei Liu +2 more
An unsupervised HMM is applied to the problem of aligning sections of a thousand policy documents, based on the issues they address, and in two new evaluations, the approach is found more effective than clustering and topic models.
Infoscience (Ecole Polytechnique Fédérale de Lausanne)Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning
45 Citations2018Hamza Harkous, Kassem Fawaz +4 more
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)A Step Towards Usable Privacy Policy: Automatic Alignment of Privacy Statements
41 Citations2018Fei Liu, Rohan Ramanath +2 more
A corpus of privacy policies of online websites is considered, and a five-point gap between system and median-human levels of agreement with a consensus annotation is suggested, of which half can be closed with bag of words representations and half requires more sophistication.
Towards an information type lexicon for privacy policies
40 Citations2015Jaspreet Bhatia, Travis D. Breaux
An information type lexicon is constructed from manual, human annotations and an entity extractor based on part-of-speech tagging that has a 31-78% chance of containing a word from any previously seen policy.
Information Security Journal A Global PerspectiveA comprehensive keyword analysis of online privacy policies
27 Citations2018Jasmin Kaur, Rozita Dara +3 more
The findings suggested that regulations have an impact on the selection of terminologies used in the privacy policies and suggested that European policies use fewer ambiguous words but use more words such as cookie and compliance with the regional regulations.
Automated Extraction of Regulated Information Types Using Hyponymy Relations
26 Citations2016Jaspreet Bhatia, Morgan C. Evans +2 more
An approach to automatically construct an information type ontology by identifying information type hyponymy in privacy policies using Tregex patterns, a utility to match regular expressions against constituency parse trees, to automate the extraction of hyponyms from privacy policies.
RECIPE: Applying Open Domain Question Answering to Privacy Policies
12 Citations2018Yan Shvartzshanider, Ananth Balashankar +2 more
It is noticed that combining the results of this model with a neural dependency parser based approach yields a significantly higher F1 score of 92.35 compared to manual annotations, which indicates that future work which in-corporates signals from parsing like NLP tasks more explicitly can generalize better on out-of-domain tasks.