Investigating the effects of representation learning on exploration in on-policy reinforcement learning

Reinforcement Learning (RL) in environments with high-dimensional state spaces is challenging. This is mainly due to the amount and quality of data required to adequately understand the environment, the consequences of actions, and to figure out high-value states/actions. Finding good actions and states, especially if they are sparse and/or there are long-term dependencies, is difficult. An RL agent must explore to find them all the while utilizing what it has learned. Additionally, the complexity of state and action spaces makes it challenging to generalize learned behaviors, requiring sophisticated function approximators and often leading to issues such as overfitting and sample inefficiency. Furthermore, the presence of noise in the data exacerbates these challenges. The effects of noise is more pronounced in high-dimensional spaces because the agent needs to discern meaningful patterns from noisy data, increasing the risk of overfitting to random fluctuations rather than true signals. Proper exploration is crucial for Reinforcement Learning problems as it can increase the sample efficiency and shorten the training time. Unguided exploration is very sample inefficient in high-dimensional settings. This is especially the case for the hard-exploration problems (e.g. Montezuma's Revenge) in which the agents struggle to learn due to the sparsity of the rewards and the complexity of the state and action spaces. There are several approaches for guided exploration, some of which are proposed to deal with the issues of hard-exploration problems. One of these methods is based on using "prediction-errors" as intrinsic rewards. In prediction-error based methods, a prediction (e.g. next state, reward) is compared against the actual observations. If the discrepancy between those two is high, one concludes that further exploration of such states is required to decrease the error. Exploration of these states is encouraged by providing extra rewards (intrinsic rewards) when the agent visits them. Such an approach adopts the optimism in the face of uncertainty principle by guiding the agent to the promising yet under-explored parts of the state space. However, in high-dimensional environments, unimportant observations and noise can lead the agent astray. One promising direction to alleviate these aforementioned issues in high-dimensional and noisy/stochastic environments is learning smaller yet effective and robust state representations. Such an ideal latent representation would be robust to noise and focus on the important aspects of the environment while ignoring the unimportant ones. Utilizing deep neural networks is already a step in this direction. Another potential step is borrowing auxiliary representation learning objectives from self-supervised learning to augment RL. In light of the observation that operating under small-dimensional state spaces is desirable for both the reinforcement learning agents and the exploration methods, we believe that for prediction-error based exploration methods, receiving support from representation learning methods appears as a viable solution. To this end we propose the Modified RND approach to investigate the effect of using an auxiliary self-supervised learning (SSL) loss for the model-predictive exploration methods. Additionally, we also propose the ViT with Explorative Attention method which aims to improve exploration performance by learning exploration and exploitation specific representations with just an architectural change without requiring any method from the self-supervised learning literature. Unfortunately, with our proposed methods we have failed to show justifiable performance gains. Only under certain circumstances we have managed to obtain better early training performance which later converged to the performance of our baseline models. Despite its short comings in empirical performance, we still believe that our work presents noteworthy ideas and serves to further one's understanding of the subject. We believe that our work may be a valuable tool to others who are also interested in the intersection of representation learning and prediction-error based on-policy exploration methods in reinforcement learning.