Comparison of recurrent deep network and vision transformer models in image classification

Image classification is a technology tool that detects certain image in images or videos and uses them in image processing. This process is also part of a series of classification processes. Various theories and methods have been put forward and applied for a long time in these classification processes made by machines. A large number of geometric, regional-based, logical, attention-based models are frequently used in various applications. Various models continue to be introduced according to needs and requirements. In recent years, image classification studies with neural networks have been increasing. In this study, it is aimed to perform image classification by using recurrent neural networks, reinforcement learning, and vision transformer. The main objective of the study is to perform the image classification process through the recurrent attention model and the vision transformer, which are two attention-based models, and to compare both attention models. The neural network compares the presented images with what it has learned before, detects the relationship between them and classifies the images. Recurrent attention models have shown superior success in different recognition and classification studies in recent years. In the study, the recurrent visual attention model is a goal-oriented tool associated with a visual environment through a limited sensor. The model is the sequential decision-making process of a goal-directed agent that engages with a visual environment through reinforcement learning. The working principle of the recurrent attention model is based on focusing only on the part of the image of interest. This saves time and reduces the number of pixels of interest. The agent in reinforcement learning can also carry out the actions and affect the real course of the environment. Since the environment is limitedly observable, the agent needs to integrate the information over time to determine what path to take and how to deploy its sensor most effectively. At each step, the agent receives a reward. The main purpose of the agent is to maximize the rewards. For the cases where the rewards are maximum, the deep network model is created. Another attention-based model used in comparison is the vision transformer. The vision transformer is a more recent model than the recurrent attention model. Transformer neural networks were first used frequently in language processing studies called NLP. In such studies, a system called BLEU score, which measures the change between automatic translation and human translation, is used and successful results are obtained. After these studies, vision transformer started to be used in various classification applications and good results were obtained in terms of time and accuracy. In the vision transformer model, the input images are first divided into fixed-size pieces, flattened and a vector is obtained. Linear embedding sequences are created from these vectors. These built-in sequences are input to the transformer encoder. In the model, there are multi-headed self-attention layer, multi-layer perceptron and norm layers. In the model, the personal attention layer has the tasks of embedding and reconstructing image information. Recurrent neural networks have two important problems. These; is the length of the training in time and the vanishing gradient problem. The length of training time can be solved with powerful GPUs and CPUs. But the vanishing gradient problem is a more important one. Problems occur in recurrent neural networks in the long term in using information from the past. Although this is partially solved through long short-term memory, it cannot provide optimum performance. Long short-term memory can act much more successfully than recurrent networks in the use of information from the past. When using transformer neural networks, information can be determined by embedding. Although the vision transformer is an attention-based model, this embedded information is stored and used by determining the priority order with different weight values. In image classification applications, the data set used to detect the presented image with superior success must go through certain stages. The study was carried out in the PyTorch library. Fashion-MNIST and CIFAR-10 datasets, which this library can access, were tested on both attention-based models and the results were compared. The CIFAR-10 dataset consists of ten different classes, including various transportation vehicles and animals. The image size is 32x32 and consists of 60000 images. The Fashion-MNIST dataset, on the other hand, is a dataset consisting of ten different clothing product types and has 70000 images. In the experiment part, Tesla T4 GPU offered by google colab to its users was used. First, training on the recurrent attention model of the Fashion-MNIST dataset was carried out. By changing the parameters such as patch size, number of glimpses, number of iterations, learning rate, which the model is highly sensitive to, its responses were measured and many trainings were carried out. Considering the image dimensions, it was deemed appropriate to have a patch size of eight. Although there were relatively small increases when the number of views was increased, it was observed that the training time increased significantly. Therefore, six was considered appropriate. Although the number of iterations was taken as 300, the training was terminated at the 256th iteration, since there was not enough improvement. The learning rate is in a decreasing form over time. When the training result graphs are examined, while training and test loss rates are decreasing; training and test accuracy rates are increasing. Afterwards, the same operations were performed on the CIFAR-10 dataset. According to the image size, the patch size was taken as twelve. The number of views has been increased to eight. The iteration and learning rate in the Fashion-MNIST dataset remained constant. Loss and accuracy rates are in line with the previous dataset. In the next step, training of the vision transformer model was carried out on Fashion-MNIST and CIFAR-10 datasets. First, in the Fashion-MNIST dataset, various trainings were carried out by changing the parameters such as patch size, depth, MLP size that the vision transformer model is sensitive to, and the results were examined. Selected parameter values are given in the table. The number of iterations was based on forty, because it was observed that the loss increased after this value. While test and training loss rates are decreasing, accuracy rates are increasing. Later, similar operations were performed on the CIFAR-10 dataset. Optimum parameter values were determined after various trials. Loss and accuracy rates are in line with the previous dataset. As a result of the training of the Fashion-MNIST dataset on the recurrent attention model, the training and test loss rates decreased; training and test accuracy rates have increased. This training process is concluded in the 256th iteration. Subsequently, as a result of training the Fashion-MNIST dataset on the vision transformer model, the training and test loss rates decreased; training and test accuracy rates have increased. In a process such as forty iterations in the vision transformer model, it approached the values reached as a result of the training carried out with the recurrent attention model. When a temporal comparison is made between the recurrent attention model and the vision transformer model of the Fashion-MNIST dataset, it is seen that the vision transformer model is more advantageous than the recurrent attention model. As a result of the training of the CIFAR-10 dataset on the recurrent attention model, the training and test loss rates decreased; training and test accuracy rates have increased. This training process concluded in the 293rd iteration. Subsequently, as a result of the training of the CIFAR-10 dataset on the vision transformer model, the training and test loss rates decreased; training and test accuracy rates have increased. In the vision transformer model, in a process such as 200 iterations, much better results were obtained than the values reached as a result of the training carried out with the recurrent attention model. When a temporal comparison is made between the recurrent attention model and the vision transformer model of the CIFAR-10 dataset, it is seen that the vision transformer model is more advantageous than the recurrent attention model. As a result, graphics and tables of both dataset training results are presented in the experiment section. According to the results obtained, it has been observed that the vision transformer model results faster in terms of training time compared to the recurrent attention model, while it gives better results in terms of accuracy and loss rates. It is predicted that better results will be obtained when pre-trained datasets are used.