Top Research Papers on Transformers

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for buil...

Abstract The narrative of ‘urban transformations’ epitomises the hope that cities provide rich opportunities for contributing to local and global sustainability and resilience. Urban transformation research is developing a rich yet consistent research agenda, offering opportunities for integrating multiple perspectives and disciplines concerned with radical change towards desirable urban systems. We outline three perspectives on urban transformations in , of and by cities as a structuring approach for integrating knowledge about urban transformations. We illustrate how each perspective helps d...

Very recently, Window-based Transformers, which computed self-attention within non-overlapping local windows, demonstrated promising results on image classification, semantic segmentation, and object detection. However, less study has been devoted to the cross-window connection which is the key element to improve the representation ability. In this work, we revisit the spatial shuffle as an efficient way to build connections among windows. As a result, we propose a new vision transformer, named Shuffle Transformer, which is highly efficient and easy to implement by modifying two lines of code....

This survey presents a taxonomy of the recent vision transformer architectures and more specifically that of the hybrid vision transformers, and sheds light on the future directions of this rapidly evolving architecture.

This work expresses the self-attention as a linear dot-product of kernel feature maps and makes use of the associativity property of matrix products to reduce the complexity from O(N) to N, where N is the sequence length.

This work proposes SortNet, as part of the Point Transformer, which induces input permutation invariance by selecting points based on a learned score, to extract local and global features and relate both representations by introducing the local-global attention mechanism.

The Point Transformer design improves upon prior work across domains and tasks and crosses the 70% mIoU threshold for the first time on the challenging S3DIS dataset.

A protein language model which takes as input a set of sequences in the form of a multiple sequence alignment and is trained with a variant of the masked language modeling objective across many protein families surpasses current state-of-the-art unsupervised structure learning methods by a wide margin.

Where racism and sexism meet—an understanding of anti-Black misogyny When Moya Bailey first coined the term misogynoir , she defined it as the ways anti-Black and misogynistic representation shape broader ideas about Black women, particularly in visual culture and digital spaces. She had no idea that the term would go viral, touching a cultural nerve and quickly entering into the lexicon. Misogynoir now has its own Wikipedia page and hashtag, and has been featured on Comedy Central’s The Daily Show and CNN’s Cuomo Prime Time . In Misogynoir Transformed , Bailey delves into her groundbreaking c...

Mit der fortschreitenden Digitalisierung gehen fundamentale gesellschaftliche Transformationsprozesse einher, die weit über eine bloße Rekonfiguration technischer, medialer oder wirtschaftlicher Verhältnisse hinausreichen. Sie betreffen sämtliche Bereiche sozialer Koordination, Kommunikation und Entwicklung der gegenwärtigen Gesellschaft. Die Einführung ordnet die viel diskutierte "digitale Transformation" in das langfristige Wechselverhältnis von Technik und Gesellschaft ein und zeigt die Diskontinuitäten, aber auch die Kontinuitäten in diesem Prozess auf. Sie gibt einen systematisierenden Üb...

Many of us in the social sciences deal with data that do not conform to assumptions of normality and/or homoscedasticity/homogeneity of variance. Some research has shown that parametric tests (e.g., multiple regression, ANOVA) can be robust to modest violations of these assumptions. Yet the reality is that almost all analyses (even nonparametric tests) benefit from improved the normality of variables, particularly where substantial non-normality is present. While many are familiar with select traditional transformations (e.g., square root, log, inverse) for improving normality, the Box-Cox tra...

This paper proposes a novel Focused Linear Attention module, which introduces a simple yet effective mapping function and an efficient rank restoration module to enhance the expressiveness of self-attention while maintaining low computation complexity.

The model of how digital transformational leadership (DTL) influences DT through organizational agility (OA) is tested, demonstrating that alignment of organizational models and evolving OA are critical to DT.

An empirically grounded conceptualization is developed that sets these two phenomena apart, finding that there are two distinctive differences: digital transformation activities leverage digital technology in (re)defining an organization’s value proposition, while IT-enabled organizational transformation activities Leverage digitalTechnology in supporting the value proposition.

A hierarchical Transformer whose representation is computed with Shifted windows, which has the flexibility to model at various scales and has linear computational complexity with respect to image size and will prove beneficial for all-MLP architectures.

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with \textbf{S}hifted \textbf{win}dows. The shifted windowing scheme brings greater efficiency by limiting self-attention ...

Transformers achieve remarkable performance in several tasks but due to their\nquadratic complexity, with respect to the input's length, they are\nprohibitively slow for very long sequences. To address this limitation, we\nexpress the self-attention as a linear dot-product of kernel feature maps and\nmake use of the associativity property of matrix products to reduce the\ncomplexity from $\\mathcal{O}\\left(N^2\\right)$ to $\\mathcal{O}\\left(N\\right)$,\nwhere $N$ is the sequence length. We show that this formulation permits an\niterative implementation that dramatically accelerates autoregre...

The transformative leadership compass is proposed as a model to outline the critical competencies distinguishing a digital transformative leader capable of driving continuous company innovation and specifically digital transformation entrepreneurship.

Abstract The impending climate emergency, the Paris agreement and Sustainable Development Goals demand significant transformations in economies and societies. Science funders, innovation agencies, and scholars have explored new rationales and processes for policymaking, such as transformative innovation policy (TIP). Here, we address the question of how to orient the efforts of science, technology, and innovation policy actors to enable transformations. We build on sustainability transitions research and a 4-year co-creation journey of the TIP Consortium to present twelve transformative outcom...

It is shown that the possibility of an entire SC integration based on new technologies is still at distance and the impact of Industry 4.0 and the Digital Transformation on decision making is greatly connected to information sharing.

Surface defect detection using deep learning approaches has become a promising area of research, but the difficulty of accurately locating and segmenting various forms of defects presents a challenge for this method. Swin Transformer, as a Transformer-based model, has made significant progress in computer vision. Its performance surpasses standard CNN’s performance on most tasks, but it has drawn scant attention from industrial applications. Thus far, using CNNs for surface defect detection tends to be the most common application. To explore the extensibility of the Transformer, we seek to exp...

Surface defect detection is an extremely crucial step to ensure the quality of industrial products. Nowadays, convolutional neural networks (CNNs) based on encoder–decoder architecture have achieved tremendous success in various defect detection tasks. However, the intrinsic locality of convolution prevents them from modeling long-range interactions explicitly, making it difficult to distinguish pseudo-defects in cluttered backgrounds. Recent transformers are especially skilled at learning global image dependencies, but with limited local structural information for the refined defect location....

This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline with an introduction to fundamental concepts behind the success of Transformers, i.e., self-attention, large-scale pre-training, and bidirectional feature encoding.

The Cross-Shaped Window self-attention mechanism for computing self-Attention in the horizontal and vertical stripes in parallel that form a cross-shaped window is developed, with each stripe obtained by splitting the input feature into stripes of equal width.

Human actions have driven earth systems close to irreversible and profound change. The need to shift towards intentional transformative adaptation (ITA) is clear. Using case studies from the Transformative Adaptation Research Alliance (TARA), we explore ITA as a way of thinking and acting that is transformative in concept and objectives, but achieved through a mix of incremental and transformative co-production processes that ultimately lead to the social-ecological system being transformed. Central to ITA are social and political issues of how individuals and collectives address environmental...

We present CSWin Transformer, an efficient and effective Transformer-based backbone for general-purpose vision tasks. A challenging issue in Transformer design is that global self-attention is very expensive to compute whereas local self-attention often limits the field of interactions of each token. To address this issue, we develop the Cross-Shaped Window self-attention mechanism for computing self-attention in the horizontal and vertical stripes in parallel that form a cross-shaped window, with each stripe obtained by splitting the input feature into stripes of equal width. We provide a mat...

This paper advocates an inductive bias of locality in video Transformers, which leads to a better speed-accuracy trade-off compared to previous approaches which compute self-attention globally even with spatial-temporal factorization.

A Pop Music Transformer is built that composes Pop piano music with better rhythmic structure than existing Transformer models, when the way a musical score is converted into the data fed to a Transformer model is improved.

This article characterizes a large and thoughtful selection of recent efficiency-flavored “X-former” models, providing an organized and comprehensive overview of existing work and models across multiple domains.

The proposed Spatio-TemporAl cRoss (STAR)-transformer, which can effectively represent two cross-modal features as a recognizable vector, achieves a promising improvement in performance in comparison to previous state-of-the-art methods.

NA is a pixel-wise operation, localizing self attention to the nearest neighboring pixels, and therefore enjoys a linear time and space complexity compared to the quadratic complexity of SA, and is the first efficient and scalable sliding window attention mechanism for vision.

This survey dives into how videos are handled at the input level first and studies the architectural changes made to deal with video more efficiently, reduce redundancy, re-introduce useful inductive biases, and capture long-term temporal dynamics.

A novel transformer-based architecture for global multi-object tracking that takes a short sequence of frames as input and produces global trajectories for all objects, and seamlessly integrates into state-of-the-art large-vocabulary detectors to track any objects.

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, w...

Methane hydrate and shale gas are predicted to have substantial reserves, far beyond the sum of other fossil fuels. Using methane instead of crude oil as a building block is, thus, a very attractive strategy for synthesizing valuable chemicals. Because methane is so inert, its direct conversion needs a high activation energy and typically requires harsh reaction conditions or strong oxidants. Photocatalysis, which employs photons operated under very mild conditions, is a promising technology to reduce the thermodynamic barrier in direct methane conversion and to avoid the common issues of over...

This paper proposes a novel Transformer architecture that elegantly exploits the global semantics for self-attention learning, namely Dual Vision Transformer (Dual-ViT).

This paper presents TExt Spotting TRansformers (TESTR), a generic end-to-end text spotting framework using Transformers for text detection and recognition in the wild, and designs a bounding-box guided polygon detection process.

First Published in 1980 (English Translation) Towards a Transformation of Philosophy presents selected essays from Karl -Otto Apel's two- volume German collection that was published in 1973 under the title Transformation der Philosophie. Karl -Otto Apel's studies in philosophy and the social sciences can be said to have bridged the gap that had hitherto existed between the Anglo-Saxon traditions of analytical philosophy of language and pragmatism, and the philosophical traditions of the European continent of phenomenology, existentialism, and hermeneutics. Apel points to language as the crucia...

This work proposes a novel Image Fusion Transformer (IFT) where a transformer-based multi-scale fusion strategy that attends to both local and long-range information (or global context) in the image fusion process.

This work describes seven critical transitions that can close the massive TB diagnostic gap and enable TB programmes worldwide to recover from the pandemic setbacks.