Proximal Policy Optimization (PPO) is a popular model-free reinforcement learning algorithm, esteemed for its simplicity and efficacy. However, due to its inherent on-policy nature, its proficiency in harnessing data from disparate policies is constrained. This paper introduces a novel off-policy extension to the original PPO method, christened Transductive Off-policy PPO (ToPPO). Herein, we provide theoretical justification for incorporating off-policy data in PPO training and prudent guidelines for its safe application. Our contribution includes a novel formulation of the policy improvement lower bound for prospective policies derived from off-policy data, accompanied by a computationally efficient mechanism to optimize this bound, underpinned by assurances of monotonic improvement. Comprehensive experimental results across six representative tasks underscore ToPPO's promising performance.

Public high school in America was the product of the time of its invention, which was way back in 1821. But in this era of rapid technological change marked by artificial intelligence and robots moving into more aspects of work and social life, maybe the way teaching is done in high school needs a reboot. It is framed around the thought experiment: What would an ideal high school of the year 2040 look like? The tour guides of this imagined school of the future are two authors: Jim Tracy, a senior advisor at the nonprofit Jobs for the Future who in his career has led private K-12 schools and served as a college president; and Greg Toppo, longtime education journalist. They instead focus on how coming technological change will end up shifting the relationship between people and machines, and therefore between students and teachers.