Applications of transformer-based language models in bioinformatics: a survey

Key developments of transformer-based language models are introduced by describing the detailed structure of transformers and the common challenges, including heterogeneity of training data, computational expense and model interpretability, and opportunities in the context of bioinformatics research are identified.