[P] New Robotics environments in OpenAI Gym • r/MachineLearning

Mujoco is mostly a physics engine, and I'm willing to bet that whatever parts you're thinking of when you say it's "more" than a physics engine either exist in some form in Bullet and the rest, or aren't relevant for RL. The things you listed are engines that delegate to other projects for their physics simulation, and come with a ton of heavyweight baggage that you don't need to do RL.