An in-depth performance analysis of neural machine translation tasks

We live in the technology era and over the last decade Artificial Intelligence (AI) has been the most focused technology. It has applications in countless topics and neural machine translation (NMT) is one of the major research areas. Many big companies like Google, Facebook, Amazon, Microsoft, Yandex deploy NMT in their production systems and NMT services has become popular lately. These services need to provide some constraints, especially speed and translation quality, for user satisfaction. There has been significant amount of work on accelerating NMT models however to the best of our knowledge, there is no detailed research giving a detailed performance analysis of each step in a model. In this work, one of the state-of-the-art models the Transformer is used. It has encoder-decoder architecture with an additional attention mechanism. A microservice is implemented on top of the Transformer model and it is showed that the bottleneck is the model itself. Then, several experiments with different configuration values has been conducted and it is observed that the performance of the model is highly sensitive to the changes in these values. A detailed performance breakdown of the model in CPU and GPU show that beam search is a big source of inefficiency. So a detailed time breakdown of the beam search is obtained in order to have a better understanding. Additionally, it is observed that the beam size only affects BLEU score at word level, and not at token level. Finally, it is showed that the vocabulary size has a major role on the performance of the beam search.