Optimizing Market Making using Multi-Agent Reinforcement Learning

Abstract--In this paper, reinforcement learning is applied to the problem of optimizing market making. A multi-agent reinforcement learning framework is used to optimally place limit orders that lead to successful trades. The framework consists of two agents. The macro-agent optimizes on making the decision to buy, sell, or hold an asset. For the context of this paper, the proposed framework is applied and studied on the Bitcoin cryptocurrency market. The goal of this paper is to show that reinforcement learning is a viable strategy that can be applied to complex problems (with complex environments) such as market making. Algorithmic trading, and in particular high-frequency algorithmic trading (HFT), has gained immense popularity in the recent decade. With advances in hardware and software, algorithmic trading has rapidly become the norm. The increasing popularity of machine learning has slowly made its way to financial markets [1], where it is primarily used to predict price movements of assets. However, there are a number of challenges that these classic machine learning techniques entail: 1) Prediction time: In machine learning, model complexity can have an impact on prediction time. Many times, in the supervised learning setting, neural networks are used to make predictions. Due to the computational complexity that comes with these models, as the model complexity increases, the decision time also increases [2]. In the HFT setting, by the time the model makes a prediction, it may already be too late to take the predicted action. The problem then becomes, how can these added latency costs be incorporated into our prediction? The general rule of thumb in finance is that the historical performance of an asset does not predict the future performance of the asset, i.e. forecasting and predicting the market from historical performance alone is virtually impossible.