rQdia: Regularizing Q-Value Distributions With Image Augmentation

With a simple auxiliary loss, that equalizes these distributions via MSE, rQdia boosts DrQ and SAC on 9/ 12 and 10 /12 tasks respectively in the MuJoCo Continuous Control Suite from pixels, and Data-Efficient Rainbow on 18/ 26 Atari Arcade environments. Gains are measured in both sample efficiency and longer-term training. Human perception is invariant to and remarkably robust against many perturbations, like discoloration, obfuscation, and low exposure. On the other hand, artificial neural networks do not intrinsically carry these invariance properties, though some invariances may be induced architecturally through inductive biases like convolution, kernel rotation, and dilation. In deep reinforcement learning (RL) from pixels, an agent is tasked to learn from raw pixels and must therefore learn to visually interpret a scene. Thus, recent approaches in deep RL have turned to the self-supervision and data augmentation techniques found in computer vision.