RLOR: A Flexible Framework of Deep Reinforcement Learning for Operation Research

Reinforcement learning has been applied in operation research and has shown promise in solving large combinatorial optimization problems. However, existing works focus on developing neural network architectures for certain problems. These works lack the flexibility to incorporate recent advances in reinforcement learning, as well as the flexibility of customizing model architectures for operation research problems. In this work, we analyze the end-to-end autoregressive models for vehicle routing problems and show that these models can benefit from the recent advances in reinforcement learning with a careful re-implementation of the model architecture. In particular, we re-implemented the Attention Model and trained it with Proximal Policy Optimization in CleanRL, showing at least 8 times speed up in training time. We hereby introduce RLOR, a flexible framework for Deep Reinforcement Learning for Operation Research. We believe that a flexible framework is key to developing deep reinforcement learning models for operation research problems. The code of our work is publicly available at https://github.com/cpwan/RLOR. Pointer Network (Vinyals et al., 2015) is a milestone work of applying neural networks in combinatorial optimization problems. It enabled dynamic input size and permutation invariance of input in the neural networks. In other words, we can feed a set to a neural network. PN+RL (Bello et al., 2019) is another milestone work. It enabled training neural networks with reinforcement learning (RL) with REINFORCE algorithm, instead of requiring expensive ground truths from solvers for supervised learning. Since then, the REINFORCE algorithm (but rarely other RL algorithms) has been used in subsequent works for vehicle routing problems, including Order-invariant PN+ RL (Nazari et al., 2018), Attention Model (Kool et al., 2019), and POMO (Kwon et al., 2020).