Imagining alternative actions for improved sample efficiency during policy learning

This study explores enhancing sample efficiency in reinforcement learning by a novel mechanism that involves the imagination of alternative actions based on a learned local approximate world model. Two different novel objectives for imagined actions that are based on hypothetical rewards and hypothetical state-action value approximations are also implemented. The potential benefits of using Conditional Neural Processes (CNP) as the policy network architecture are also investigated in comparative analyses. The proposed method is tested in several simulated environments, including Continuous Cart Pole, Hopper, and Walker2d. Our findings reveal that while CNPs do not improve learning in the tested environments, the alternative action imagination mechanism shows promise for improving performance when implemented with MLPs and learned world models. In particular, this mechanism effectively guided exploration, leading to potentially faster convergence and improved average episode returns in two experiments. However, its effectiveness varied across different setups and was less pronounced in the Walker2d environment. The study also highlighted the risks of over-optimization when using perfect world models, emphasizing the benefits of learned models in maintaining effective exploration and preventing premature convergence to sub-optimal policies.