CLEVE: Contrastive Pre-training for Event Extraction
Published 1 January 2021Open access
Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu
Citations106
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
CLEVE is proposed, a contrastive pre-training framework for EE to better learn event knowledge from large unsupervised data and their semantic structures (e.g. AMR) obtained with automatic parsers.
Abstract
Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li, Jie Zhou. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)MizAR 60 for Mizar 50
76,311 Citations2023Jakubův, Jan, Chvalovský, Karel +7 more
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
18,050 Citations2013Tomáš Mikolov, Kai Chen +2 more
Two novel model architectures for computing continuous vector representations of words from very large data sets are proposed and it is shown that these vectors provide state-of-the-art performance on the authors' test set for measuring syntactic and semantic word similarities.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case
17,334 Citations2019Yinhan Liu, Myle Ott +8 more
This work considers the task of building machine learning models to automatically select the best combination for a problem instance and contributes to the automatic learning of instance features directly from the high-level representation of a problem instance using a transformer encoder.
Momentum Contrast for Unsupervised Visual Representation Learning
12,039 Citations2020Kaiming He, Haoqi Fan +3 more
Statistics and ComputingA tutorial on spectral clustering
10,269 Citations2007Ulrike von Luxburg
This tutorial describes different graph Laplacians and their basic properties, present the most common spectral clustering algorithms, and derive those algorithms from scratch by several different approaches.
arXiv (Cornell University)A Simple Framework for Contrastive Learning of Visual Representations
7,335 Citations2020Ting Chen, Simon Kornblith +2 more
It is shown that composition of data augmentations plays a critical role in defining effective predictive tasks, and introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning.
Dimensionality Reduction by Learning an Invariant Mapping
5,057 Citations2006Raia Hadsell, Sumit Chopra +1 more
This work presents a method - called Dimensionality Reduction by Learning an Invariant Mapping (DrLIM) - for learning a globally coherent nonlinear function that maps the data evenly to the output manifold.
arXiv (Cornell University)Representation Learning with Contrastive Predictive Coding
4,488 Citations2018Aäron van den Oord, Yazhe Li +1 more
arXiv (Cornell University)Representation Learning with Contrastive Predictive Coding
3,769 Citations2018Aäron van den Oord, Yazhe Li +1 more
This work proposes a universal unsupervised learning approach to extract useful representations from high-dimensional data, which it calls Contrastive Predictive Coding, and demonstrates that the approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
Unsupervised Feature Learning via Non-parametric Instance Discrimination
3,510 Citations2018Zhirong Wu, Yuanjun Xiong +2 more
This work forms this intuition as a non-parametric classification problem at the instance-level, and uses noise-contrastive estimation to tackle the computational challenges imposed by the large number of instance classes.
Zenodo (CERN European Organization for Nuclear Research)Formal Semantics for Kolmogorov-Arnold Network Representations of Operational Games
3,475 Citations2025Keyulu Xu
This work characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures, and develops a simple architecture that is provably the most expressive among the class of GNNs.
Dagstuhl Research Online Publication ServerIdentifying Resilient Communities in Road Networks: A Path-Based Embedding Approach
1,607 Citations2025Wagner, Christopher, Dodge, Somayeh +1 more
Leibniz-Zentrum für Informatik (Schloss Dagstuhl)Question Answering For Toxicological Information Extraction
1,552 Citations2022Kevin Clark, Minh-Thang Luong +3 more
This work proposes an approach that relies on Question Answering for acquiring information from unstructured data, in this case, English PDF documents containing information about physicochemical and toxicological properties of chemical substances.
arXiv (Cornell University)Learning deep representations by mutual information estimation and maximization
1,390 Citations2018R Devon Hjelm, Alex Fedorov +5 more
It is shown that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation’s suitability for downstream tasks and is an important step towards flexible formulations of representation learning objectives for specific end-goals.
TIB Data ManagerA simple framework for contrastive learning of visual representations
1,203 Citations2024Ting Chen
Abstract Meaning Representation for Sembanking
1,131 Citations2013Laura Banarescu, Claire Bonial +8 more
A sembank of simple, whole-sentence semantic structures will spur new work in statistical natural language understanding and generation, like the Penn Treebank encouraged work on statistical parsing.
Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks
929 Citations2015Yubo Chen, Liheng Xu +3 more
A word-representation model to capture meaningful semantic regularities for words and a framework based on a convolutional neural network to capture sentence-level clues are introduced.
arXiv (Cornell University)Variational Graph Auto-Encoders
895 Citations2016Thomas Kipf, Max Welling
The variational graph auto-encoder (VGAE) is introduced, a framework for unsupervised learning on graph-structured data based on the variational auto- Encoder (VAE) that can naturally incorporate node features, which significantly improves predictive performance on a number of benchmark datasets.
arXiv (Cornell University)Graph Contrastive Learning with Augmentations
858 Citations2020Yuning You, Tianlong Chen +4 more
The results show that, even without tuning augmentation extents nor using sophisticated GNN architectures, the GraphCL framework can produce graph representations of similar or better generalizability, transferrability, and robustness compared to state-of-the-art methods.
Information RetrievalA comparison of extrinsic clustering evaluation metrics based on formal constraints
839 Citations2008Enrique Amigó, Julio Gonzalo +2 more
This article defines a few intuitive formal constraints on such metrics which shed light on which aspects of the quality of a clustering are captured by different metric families, and proposes a modified version of Bcubed that avoids the problems found with other metrics.
Joint Event Extraction via Recurrent Neural Networks
665 Citations2016Thien Huu Nguyen, Kyunghyun Cho +1 more
This work proposes to do event extraction in a joint framework with bidirectional recurrent neural networks, thereby benefiting from the advantages of the two models as well as addressing issues inherent in the existing approaches.
Entity-based cross-document coreferencing using the Vector Space Model
533 Citations1998Amit Bagga, Breck Baldwin
A cross-document coreference resolution algorithm which uses the Vector Space Model to resolve ambiguities between people having the same name and this algorithm is compared to the scoring algorithm used in the MUC-6 (within document) coreference task.
The stages of event extraction
511 Citations2006David Ahn
This paper presents a simple, modular approach to event extraction that allows for experiment with a variety of machine learning methods for sub-tasks of varying difficulty, as well as to evaluate the impact on performance these sub-Tasks have on the overall task.
Joint Event Extraction via Structured Prediction with Global Features
478 Citations2013Qi Li, Heng Ji +1 more
This work proposes a joint framework based on structured prediction which extracts triggers and arguments together so that the local predictions can be mutually improved, and proposes to incorporate global features which explicitly capture the dependencies of multiple triggers and argued.
Event Detection and Domain Adaptation with Convolutional Neural Networks
416 Citations2015Thien Huu Nguyen, Ralph Grishman
Convolutional neural networks that overcome the two fundamental limitations of the traditional feature-based approaches to this task: complicated feature engineering for rich feature sets and error propagation from the preceding stages which generate these features are studied.
arXiv (Cornell University)Deep Graph Contrastive Representation Learning
408 Citations2020Yanqiao Zhu, Yichen Xu +4 more
This paper proposes a novel framework for unsupervised graph representation learning by leveraging a contrastive objective at the node level, and generates two graph views by corruption and learns node representations by maximizing the agreement of node representations in these two views.
Refining Event Extraction through Cross-Document Inference
400 Citations2008Heng Ji, Ralph Grishman
This work designs a simple scheme to conduct cross-document inference for improving the ACE event extraction task, and applies the hypothesis of “One Sense Per Discourse” to information extraction.
Proceedings of the AAAI Conference on Artificial IntelligenceGraph Convolutional Networks With Argument-Aware Pooling for Event Detection
386 Citations2018Thien Huu Nguyen, Ralph Grishman
This work investigates a convolutional neural network based on dependency trees to perform event detection and proposes a novel pooling method that relies on entity mentions to aggregate the convolution vectors.
Exploring Pre-trained Language Models for Event Extraction and Generation
343 Citations2019Sen Yang, Dawei Feng +3 more
This work proposes an event extraction model to overcome the roles overlap problem by separating the argument prediction in terms of roles, and proposes a method to automatically generate labeled data by editing prototypes and screen out generated samples by ranking the quality.
Knowledge Base Population: Successful Approaches and Challenges
311 Citations2011Heng Ji, Ralph Grishman
The techniques which can serve as a basis for a good KBP system are provided, the remaining challenges by comparison with traditional Information Extraction (IE) and Question Answering (QA) tasks are laid out, and some suggestions to address these challenges are provided.
Event Extraction as Machine Reading Comprehension
303 Citations2020Jian Liu, Yubo Chen +3 more
This paper proposes a new learning paradigm of EE, by explicitly casting it as a machine reading comprehension problem (MRC), which includes an unsupervised question generation process, which can transfer event schema into a set of natural questions, followed by a BERT-based question-answering process to retrieve answers as EE results.
Using Document Level Cross-Event Inference to Improve Event Extraction
296 Citations2010Shasha Liao, Ralph Grishman
This paper uses document level information from the training corpus to improve the performance of ACE event extraction and uses information about other types of events to make predictions or resolve ambiguities regarding a given event.
Proceedings of the AAAI Conference on Artificial IntelligenceJointly Extracting Event Triggers and Arguments by Dependency-Bridge RNN and Tensor-Based Argument Interaction
255 Citations2018Lei Sha, Feng Qian +2 more
A novel dependency bridge recurrent neural network (dbRNN) is proposed that simultaneously applying tree structure and sequence structure in RNN brings much better performance than only uses sequential RNN.
Learning from Context or Names? An Empirical Study on Neural Relation Extraction
183 Citations2020Hao Peng, Tianyu Gao +6 more
An entity-masked contrastive pre-training framework for RE is proposed to gain a deeper understanding on both textual context and type information while avoiding rote memorization of entities or use of superficial cues in mentions.
Zero-Shot Transfer Learning for Event Extraction
178 Citations2018Lifu Huang, Heng Ji +4 more
A transferable architecture of structural and compositional neural networks is designed to jointly represent and map event mentions and types into a shared semantic space and can select, for each event mention, the event type which is semantically closest in this space as its type.
Automatically Labeled Data Generation for Large Scale Event Extraction
172 Citations2017Yubo Chen, Shulin Liu +3 more
This work proposes to automatically label training data for event extraction via world knowledge and linguistic knowledge, which can detect key arguments and trigger words for each event type and employ them to label events in texts automatically.
Template-Based Information Extraction without the Templates
172 Citations2011Nathanael Chambers, Dan Jurafsky
This paper describes an approach to template-based IE that removes this requirement and performs extraction without knowing the template structure in advance, and induces template structure very similar to hand-created gold structure from raw text.
Adversarial Training for Weakly Supervised Event Detection
150 Citations2019Xiaozhi Wang, Xu Han +3 more
The experiments show that the candidate selection and adversarial training can cooperate together to obtain more diverse and accurate training data for ED, and significantly outperform the state-of-the-art methods in various weakly supervised scenarios.
Event Detection with Multi-Order Graph Convolution and Aggregated Attention
145 Citations2019Haoran Yan, Xiaolong Jin +3 more
A new method for event detection is proposed, which uses a dependency tree based graph convolution network with aggregative attention to explicitly model and aggregate multi-order syntactic representations in sentences.
MAVEN: A Massive General Domain Event Detection Dataset
143 Citations2020Xiaozhi Wang, Ziqi Wang +8 more
A MAssive eVENt detection dataset is presented, which contains 4,480 Wikipedia documents, 117,200 event mention instances, and 207 event types and shows that existing ED methods cannot achieve promising results as on the small datasets, which suggests ED in real world remains a challenging task and requires further research efforts.
Neural Information Processing SystemsGraph Contrastive Learning with Augmentations
131 Citations2020Yuning You, Tianlong Chen +4 more
Event Schema Induction with a Probabilistic Entity-Driven Model
123 Citations2013Nathanael Chambers
This paper presents the first generative model for schema induction that integrates coreference chains into learning, and matches the pipeline’s performance, and outperforms the HMM by 7 F1 points.
Improving Event Detection via Open-domain Trigger Knowledge
106 Citations2020Meihan Tong, Bin Xu +5 more
A novel Enrichment Knowledge Distillation (EKD) model to leverage external open-domain trigger knowledge to reduce the in-built biases to frequent trigger words in annotations is proposed.
HMEAE: Hierarchical Modular Event Argument Extraction
94 Citations2019Xiaozhi Wang, Ziqi Wang +7 more
A Hierarchical Modular Event Argument Extraction model is proposed, to provide effective inductive bias from the concept hierarchy of event argument roles and significantly outperform the state-of-the-art baselines.
Expert Systems with ApplicationsEvent graphs for information retrieval and multi-document summarization
92 Citations2014Goran Glavaš, Jan Šnajder
This article presents event graphs, a novel event-based document representation model that filters and structures the information about events described in text that significantly outperforms well-established retrieval models on event-oriented test collections, and outperforms competitive models from shared multi-document summarization tasks.
A Comparison of the Events and Relations Across ACE, ERE, TAC-KBP, and FrameNet Annotation Standards
89 Citations2014Jacqueline Aguilar, Charley Beller +5 more
An overview of the relationship between ACE and ERE and compares them to the more restricted standard of the TACKBP slot-filling task and the more expansive standard of FrameNet is provided.
arXiv (Cornell University)A Mutual Information Maximization Perspective of Language Representation Learning
81 Citations2019Lingpeng Kong, Cyprien de Masson d’Autume +4 more
This work shows state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word sequence (i.e., a sentence).
Event Detection: Gate Diversity and Syntactic Importance Scores for Graph Convolution Neural Networks
77 Citations2020Viet Dac Lai, Tuan Ngo Nguyen +1 more
This study proposes a novel gating mechanism to filter noisy information in the hidden vectors of the GCN models for ED based on the information from the trigger candidate and introduces novel mechanisms to achieve the contextual diversity for the gates and the importance score consistency for the graphs and models in ED.
Predicting unknown time arguments based on cross-event propagation
76 Citations2009Prashant Gupta, Heng Ji
Two methods are described, one based on rules and the other based on statistical learning, to predict the unknown time argument for an event by the propagation from its related events.
Boosting Transition-based AMR Parsing with Refined Actions and Auxiliary Analyzers
75 Citations2015Chuan Wang, Nianwen Xue +1 more
Improved AMR parsing results are reported by adding a new action to a transitionbased AMR parser to infer abstract concepts and by incorporating richer features produced by auxiliary analyzers such as a semantic role labeler and a coreference resolver.
Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
73 Citations2020Dan Iter, Kelvin Guu +2 more
Conpono, an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences is proposed, and it is shown that Conpono yields gains of 2%-6% absolute even for tasks that do not explicitly evaluate discourse: textual entailment, common sense reasoning and reading comprehension.
Generative Event Schema Induction with Entity Disambiguation
65 Citations2015Kiem-Hieu Nguyen, Xavier Tannier +2 more
This paper presents a generative model to event schema induction that precisely represents elements other than head words using probabilistic topic distributions and illustrates that such information plays an important role in parameter estimation.
Improving AMR Parsing with Sequence-to-Sequence Pre-training
57 Citations2020Dongqin Xu, Junhui Li +3 more
This paper proposes a seq2seq pre-training approach to build pre-trained models in both single and joint way on three relevant tasks, i.e., machine translation, syntactic parsing, and AMR parsing itself, and extends the vanilla fine-tuning method to a multi-task learning fine- Tuning method that optimizes for the performance of AMR parse while endeavors to preserve the response of pre- trained models.
Semi-supervised New Event Type Induction and Event Detection
51 Citations2020Lifu Huang, Heng Ji
A Semi-Supervised Vector Quantized Variational Autoencoder framework is designed to automatically learn a discrete latent type representation for each seen and unseen type and optimize them using seen type event annotations.
arXiv (Cornell University)A Mutual Information Maximization Perspective of Language Representation Learning
46 Citations2020Lingpeng Kong, Cyprien de Masson d’Autume +4 more
Train No Evil: Selective Masking for Task-Guided Pre-Training
43 Citations2020Yuxian Gu, Zhengyan Zhang +3 more
Experimental results on two sentiment analysis tasks show that the proposed selective masking task-guided pre-training method can achieve comparable or even better performance with less than 50\% overall computation cost, which indicates the method is both effective and efficient.
Joint Learning Templates and Slots for Event Schema Induction
30 Citations2016Lei Sha, Sujian Li +2 more
A joint entity-driven model to learn template and slots simultaneously based on the constraints of templates and slots in the same sentence is proposed and the entities' semantic information is also considered for the inner connectivity of the entities.
QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization
25 Citations2019Yi-Ting Yeh, Yun-Nung Chen
This work proposes QAInfomax as a regularizer in reading comprehension systems by maximizing mutual information among passages, a question, and its answer to help regularize the model to not simply learn the superficial correlation for answering the questions.
Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning
17 Citations2020Wanyun Cui, Guangyu Zheng +1 more
The proposed Multimodal Aligned Contrastive Decoupled learning network forces the decoupled text encoder to represent the visual information via contrastive learning, and embeds visual knowledge even for plain text inference.