Deep learning methods with pre-trained word embeddings and pre-trained transformers for extreme multi label text classification

In recent years, there appears to be a remarkable increase int the number of textual documents online. This increase requires the creation of highly improved machine learning methods to classify text in many different domains. The effectiveness of these machine learning methods depends on the model capacity to understand the complex nature of the unstructured data and the relations of the features that exist. Many different machine learning methods were proposed for a long time to solve text classification problems, such as SVM, kNN and Rocchio classification. These shallow learning methods have achieved doubtless success in many different domains. For big and unstructured data like text, deep learning methods which can learn representations and features from the input data without using any feature extraction methods have shown to be one of the major solutions. In this study, we explore the accuracy of recent recommended deep learning methods for multi-label text classification starting with simple RNN, CNN models to pre-trained transformer models. We evaluated these methods' performances by computing multi-label evaluation metrics and compared the results with the previous studies.