Efficient Nonlinear Control with Actor-Tutor Architecture

Itwas demonstrated in the simulation of a pendulum swing-up task that the value-gradient based control scheme requires much less learning trials than the conventional "actor-critic"control scheme (Barto et al., 1983). In the actor-critic scheme, the actor, a direct feedback controller, improves its control policystochastically using the TD error as the effective reinforcement (Figure 1a). Despite its relatively slow learning, the actor-critic architecture has the virtue of simple computation in generating control command. In order to train a direct controller while making efficient use of the value function, we propose a new reinforcement learning scheme which we call the "actor-tutor" architecture (Figure 1b).