Transformer to CNN: Label-scarce distillation for efficient text\n classification

Abstract

Significant advances have been made in Natural Language Processing (NLP)\nmodelling since the beginning of 2018. The new approaches allow for accurate\nresults, even when there is little labelled data, because these NLP models can\nbenefit from training on both task-agnostic and task-specific unlabelled data.\nHowever, these advantages come with significant size and computational costs.\nThis workshop paper outlines how our proposed convolutional student\narchitecture, having been trained by a distillation process from a large-scale\nmodel, can achieve 300x inference speedup and 39x reduction in parameter count.\nIn some cases, the student model performance surpasses its teacher on the\nstudied tasks.\n