Investigating the missing pieces of sensorimotor reinforcement learning agents for autonomous driving

Reinforcement Learning (RL) has the potential to surpass human capabilities in self-driving without needing any expert supervision. Despite its promise, the state-of-the-art in sensorimotor self-driving is dominated by imitation learning methods due to the inherent challenges of RL algorithms. Nonetheless, RL agents are able to discover highly successful policies when provided with privileged ground truth representations of the environment. In this work, we investigate what separates privileged RL agents from sensorimotor agents for urban driving in order to bridge the gap between the two. We propose vision-based deep learning models to approximate the privileged representations from sensor data. In particular, we identify aspects of state representation that are crucial for the success of the RL agent such as desired route generation and traffic light prediction, and propose solutions to gradually remove privileged information for each with the existing computer vision approaches. Through rigorous evaluation on the CARLA simulation, we shed light on the significance of the state representation in RL for autonomous driving and outline unresolved challenges for future research.