Appendix A Application of mixreg to PPO and Rainbow