Lessons Learned Reproducing a Deep Reinforcement Learning Paper

There are a lot of neat things going on in deep reinforcement learning. One of the coolest things from last year was OpenAI and DeepMind's work on training an agent using feedback from a human rather than a classical reward signal. There's a great blog post about it at Learning from Human Preferences, and the original paper is at Deep Reinforcement Learning from Human Preferences. I've seen a few recommendations that reproducing papers is a good way of levelling up machine learning skills, and I decided this could be an interesting one to try with. It was indeed a super fun project, and I'm happy to have tackled it - but looking back, I realise it wasn't exactly the experience I thought it would be. If you're thinking about reproducing papers too, here are some notes on what surprised me about working with deep RL.