How to Train good Word Embeddings for Biomedical NLP
Published 1 January 2016Open access
Billy Chiu, Gamal Crichton, Anna Korhonen, Sampo Pyysalo
Citations352
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A comprehensive study of how the quality of embeddings changes according to hyper-parameter settings is presented, and it is observed that bigger corpora do not necessarily produce better biomedical domain word embedDings.
Abstract
The quality of word embeddings depends on the input corpora, model architectures, and hyper-parameter settings. Using the state-of-the-art neural embedding tool word2vec and both intrinsic and extrinsic evaluations, we present a comprehensive study of how the quality of embeddings changes according to these features. Apart from identifying the most influential hyper-parameters, we also observe one that creates contradictory results between intrinsic and extrinsic evaluations. Furthermore, we find that bigger corpora do not necessarily produce better biomedical domain word embeddings. We make our evaluation tools and resources as well as the created state-of-the-art word embeddings available under open licenses from https://github.com/ cambridgeltl/BioNLP-2016.
Keywords
Computer ScienceBiochemistry, Genetics and Molecular Biology
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
arXiv (Cornell University)Distributed Representations of Words and Phrases and their Compositionality
18,086 Citations2013Tomáš Mikolov, Ilya Sutskever +3 more
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
arXiv (Cornell University)Efficient Estimation of Word Representations in Vector Space
18,050 Citations2013Tomáš Mikolov, Kai Chen +2 more
Two novel model architectures for computing continuous vector representations of words from very large data sets are proposed and it is shown that these vectors provide state-of-the-art performance on the authors' test set for measuring syntactic and semantic word similarities.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
NLTK
3,299 Citations2002Edward Loper, Steven Bird
The Natural Language Toolkit has been rewritten, simplifying many linguistic data structures and taking advantage of recent enhancements in the Python language.
Journal of Machine Learning ResearchA neural probabilistic language model
2,663 Citations2003BengioYoshua, DucharmeRéjean +2 more
This work reports on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach very significantly improves on a state-of-the-art trigram model.
Word Representations: A Simple and General Method for Semi-Supervised Learning
1,944 Citations2010Joseph Turian, Lev-Arie Ratinov +1 more
This work evaluates Brown clusters, Collobert and Weston (2008) embeddings, and HLBL (Mnih & Hinton, 2009) embeds of words on both NER and chunking, and finds that each of the three word representations improves the accuracy of these baselines.
arXiv (Cornell University)NLTK: The Natural Language Toolkit
1,935 Citations2002Edward Loper, Steven Bird
Transactions of the Association for Computational LinguisticsImproving Distributional Similarity with Lessons Learned from Word Embeddings
1,342 Citations2015Omer Levy, Yoav Goldberg +1 more
It is revealed that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves, and these modifications can be transferred to traditional distributional models, yielding similar gains.
Computational LinguisticsSimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation
1,197 Citations2015Felix Hill, Roi Reichart +1 more
SimLex-999 is presented, a gold standard resource for evaluating distributional semantic models that improves on existing resources in several important ways, and explicitly quantifies similarity rather than association or relatedness so that pairs of entities that are associated but not actually similar have a low rating.
Introduction to the bio-entity recognition task at JNLPBA
579 Citations2004Jin-Dong Kim, Tomoko Ohta +3 more
The JNLPBA shared task of bio-entity recognition using an extended version of the GENIA version 3 named entity corpus of MEDLINE abstracts is described and a general discussion of the approaches taken by participating systems is presented.
Genome biologyOverview of BioCreative II gene mention recognition
490 Citations2008Larry Smith, Lorraine Tanabe +32 more
It is demonstrated that, by combining the results from all submissions, an F score of 0.9066 is feasible, and furthermore that the best result makes use of the lowest scoring submissions.
Research Portal (King's College London)Distributional Semantics Resources for Biomedical Text Processing
441 Citations2013Sampo Pyysalo, Filip Ginter +3 more
This study introduces the first set of such language resources created from analysis of the entire available biomedical literature, including a dataset of all 1to 5-grams and their probabilities in these texts and new models of word semantics.
PubMedSemantic Similarity and Relatedness between Clinical Terms: An Experimental Study.
173 Citations2010Serguei Pakhomov, Bridget T. McInnes +4 more
The results of the study confirm the existence of a measurable mental representation of semantic relatedness between medical terms that is distinct from similarity and independent of the context in which the terms occur.
Transactions of the Association for Computational LinguisticsA Large Scale Evaluation of Distributional Semantic Models: Parameters, Interactions and Model Selection
155 Citations2014Gabriella Lapesa, Stefan Evert
This study combines a broad coverage of model parameters with a model selection methodology that is robust to overfitting and able to capture parameter interactions and shows that the strategy allows us to identify parameter configurations that achieve good performance across different datasets and tasks.
Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance
97 Citations2016Billy Chiu, Anna Korhonen +1 more
It is demonstrated that most intrinsic evaluations are poor predictors of downstream performance, and this issue can be traced in part to a failure to distinguish specific similarity from relatedness in intrinsic evaluation datasets.
Evaluating distributed word representations for capturing semantics of biomedical concepts
91 Citations2015MUNEEB TH, Sunil Kumar Sahu +1 more
This work aims to compare the performance of two state-of-the-art word embedding methods, namely word2vec and GloVe on a basic task of reflecting semantic similarity and relatedness of biomedical concepts.
GPLSI: Supervised Sentiment Analysis in Twitter using Skipgrams
29 Citations2014Javi Fernández, Yoan Gutiérrez +2 more
The novelty of the approach lies in the use of words, ngrams and skipgrams as features, and how they are weighted.