Developing intelligent agents playing video games with deep Q-learning and examining the effect of experience replay buffer size on training

Reinforcement learning is based on the use of experiences obtained from interacting with the environment, instead of a ready-made dataset, unlike supervised and unsupervised learning. Q-learning and many other reinforcement learning algorithms aim to obtain the optimal action-value function by using these experiences in iterative updates. Although such classical reinforcement learning methods have shown success in environments with low-dimensional state space, they are not practical to use in complex problems involving high-dimensional state space, such as playing video games. Using neural networks as function approximators for such problems allows generalization about states that are not observed during training. However, since the agent's experiences are sequential and there is a high correlation between consecutive samples, training the neural network in this way will lead to catastrophic forgetting. To prevent this situation, an independent and identical distribution is obtained by storing the experiences in a buffer and using randomly selected samples. This method, called experience replay, makes the training stable by breaking the correlation and increasing data efficiency. Although it is known that the replay buffer is a critical mechanism for many deep reinforcement learning algorithms, it is seen that the size of the buffer is not given enough importance in the literature. In this study, agents were developed to play various Atari games using the deep Q-learning algorithm with replay buffers of three different sizes: 50 000, 100 000, and 150 000, and the effect of the buffer size on training was observed. When the results obtained from 90 agents developed for ten different games are examined, it is seen that human performance is exceeded in six games, human-level performance is achieved in two games, and low success is achieved in two games. In terms of experience replay, while increasing the buffer size continuously increased success in five games, it caused a continuous decrease in success in two games. The results show that using a larger replay buffer does not always increase succcess, and the buffer size is an important hyperparameter that needs to be tuned.