Comprehensive analysis of transformer model components in natural language processing

The Transformer architecture, based entirely on attention mechanisms, has eliminated the need for recurrent and convolutional models and has formed the foundation of advanced language models such as ChatGPT. Nevertheless, compared to traditional deep learning approaches, the Transformer incorporates a large number of components and layers, which makes comprehensive analysis a challenging research task requiring high computational costs and long training times. To address these challenges, this study considers the Transformer architecture across three distinct stages. In the first stage, the Transformer architecture is addressed from a holistic perspective, and the effects of individual components as well as component pairs on model performance are systematically analyzed. In the second stage, while the core architectural components identified in the initial stage were kept under control, the performance of various optimization algorithms is analyzed with a higher resolution in the context of their interaction with model complexity and dataset scale. In the final stage, the study focuses on the critical role of the Key and Value dimensions within the multi-head attention mechanism. The results demonstrate that the contributions of architectural components to model performance are not homogeneous and that certain components and component pairs exhibit more dominant effects than others. These findings indicate that not all components exert equal influence in architectural design and hyperparameter optimization processes, either individually or through their interactions with other hyperparameters. In this context, accurately identifying the dominant effects enables high performance to be achieved with lower parameter configurations, leading to substantial reductions in both computational cost and training time