An Empirical Study of Pre-trained Transformers for Arabic Information Extraction
Published 1 January 2020Open access
Wuwei Lan, Chen Yang, Wei Xu, Alan Ritter
Citations66
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper pre-train a customized bilingual BERT that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning, and studies GigaBERT's effectiveness on zero-short transfer across four IE tasks: named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction.
Abstract
Multilingual pre-trained Transformers, such as mBERT However, their performance on Arabic information extraction (IE) tasks is not very well studied. In this paper, we pre-train a customized bilingual BERT, dubbed GigaBERT, that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning. We study Giga-BERT's effectiveness on zero-short transfer across four IE tasks: named entity recognition, part-of-speech tagging, argument role labeling, and relation extraction. Our best model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT (Antoun et al., 2020) in both the supervised and zero-shot transfer settings.
Keywords
Computer Science
UvA-DARE (University of Amsterdam)Adam: A Method for Stochastic Optimization
84,783 Citations2014Diederik P. Kingma, Jimmy Ba
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)MizAR 60 for Mizar 50
76,311 Citations2023Jakubův, Jan, Chvalovský, Karel +7 more
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)
50,318 Citations2021Mandi, Jayanta, Canoy, Rocsildes +2 more
A simple numeric simulation of DNA-co-polymerized hydrogel shape change and a genetic algorithm that generates and selects large batches of material designs that compete with one another to evolve and converge on optimal objective-matching designs are constructed.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at Scale
45,681 Citations2018Kefallinos, Dionysios, Alexandris, Georgios +6 more
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)HISTORIAE, History of Socio-Cultural Transformation as Linguistic Data Science. A Humanities Use Case
17,334 Citations2019Yinhan Liu, Myle Ott +8 more
This work considers the task of building machine learning models to automatically select the best combination for a problem instance and contributes to the automatic learning of instance features directly from the high-level representation of a problem instance using a transformer encoder.
arXiv (Cornell University)PyTorch: An Imperative Style, High-Performance Deep Learning Library
16,158 Citations2019Adam Paszke, Sam Gross +19 more
The Stanford CoreNLP Natural Language Processing Toolkit
7,238 Citations2014Christopher D. Manning, Mihai Surdeanu +4 more
The design and use of the Stanford CoreNLP toolkit is described, an extensible pipeline that provides core natural language analysis, and it is suggested that this follows from a simple, approachable design, straightforward interfaces, the inclusion of robust and good quality analysis components, and not requiring use of a large amount of associated baggage.
arXiv (Cornell University)Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
5,634 Citations2016Yonghui Wu, Mike Schuster +29 more
GNMT, Google's Neural Machine Translation system, is presented, which attempts to address many of the weaknesses of conventional phrase-based translation systems and provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delicited models.
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
2,471 Citations2019Myle Ott, Sergey Edunov +6 more
Fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks and supports distributed training across multiple GPUs and machines.
arXiv (Cornell University)Cross-lingual Language Model Pretraining
1,616 Citations2019Guillaume Lample, Alexis Conneau
This work proposes two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingsual language model objective.
arXiv (Cornell University)AraBERT: Transformer-based Model for Arabic Language Understanding
614 Citations2020Wissam Antoun, Fady Baly +1 more
This paper pre-trained BERT specifically for the Arabic language in the pursuit of achieving the same success that BERT did for the English language, and showed that the newly developed AraBERT achieved state-of-the-art performance on most tested Arabic NLP tasks.
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
594 Citations2019Shijie Wu, Mark Dredze
This paper explores the broader cross-lingual potential of mBERT (multilingual) as a zero shot language transfer model on 5 NLP tasks covering a total of 39 languages from various language families: NLI, document classification, NER, POS tagging, and dependency parsing.
Publication Server of the Institute for German Language (Institute for German Language)Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures
236 Citations2019Pedro Ortiz Suárez, Benoît Sagot +1 more
A general, highly parallel, multithreaded pipeline to clean and classify Common Crawl by language is proposed and developed so that it runs efficiently on medium to low resource infrastructures where I/O speeds are the main constraint.
Joint Mention Extraction and Classification with Mention Hypergraphs
230 Citations2015Wei Lu, Dan Roth
A novel model is presented for the task of joint mention extraction and classification that is able to effectively capture overlapping mentions with unbounded lengths and can be extended to additionally capture mention heads explicitly in a joint manner under the same time complexity.
Emerging Cross-lingual Structure in Pretrained Language Models
208 Citations2020Alexis Conneau, Shijie Wu +3 more
It is shown that transfer is possible even when there is no shared vocabulary across the monolingual corpora and also when the text comes from very different domains, and it is strongly suggested that, much like for non-contextual word embeddings, there are universal latent symmetries in the learned embedding spaces.
arXiv (Cornell University)Cross-Lingual Ability of Multilingual BERT: An Empirical Study
172 Citations2019K Karthikeyan, Zihan Wang +2 more
arXiv (Cornell University)AraBERT: Transformer-based Model for Arabic Language Understanding
135 Citations2020Wissam Antoun, Fady Baly +1 more
Neural CRF Model for Sentence Alignment in Text Simplification
103 Citations2020Chao Jiang, Mounica Maddela +3 more
A novel neural CRF alignment model is proposed which not only leverages the sequential nature of sentences in parallel documents but also utilizes a neural sentence pair model to capture semantic similarity.
hULMonA: The Universal Language Model in Arabic
68 Citations2019Obeida ElJundi, Wissam Antoun +4 more
Developing the first Universal Language Model in Arabic (hULMonA) and demonstrating its use for Arabic classifications tasks, and demonstrating how a pre-trained multi-lingual BERT can also be used for Arabic.
QE BERT: Bilingual BERT Using Multi-task Learning for Neural Quality Estimation
31 Citations2019Hyun Kim, Joonho Lim +2 more
This proposed model is re-purposed BERT for the translation quality estimation and uses multi-task learning for the sentence-level task and word-level subtasks (i.e., source word, target word, and target gap).