MT-GAN-BERT: Multi-Task and Generative Adversarial Learning for sustainable Language Processing

In this paper, we present MT-GAN-BERT, i.e., a BERT-based architecture for faceted classification tasks. It aims to reduce the requirements of Transformers both in terms of the amount of annotated data and the computational cost required at classification time. First, MT-GAN-BERT enables semi-supervised learning in BERT-based architectures based on Generative Adversarial Learning. Second, it implements a Multi-task Learning approach to solve multiple tasks simultaneously. A single BERTbased model is used to encode the input examples, while multiple linear layers are used to implement the classification steps, with a significant reduction of the computational costs. Experimental evaluations against six classification tasks involved in detecting abusive languages in Italian suggest that MT-GAN-BERT represents a sustainable solution that generally improves the raw adoption of multiple BERT-based models with lighter requirements in terms of annotated data and computational costs.