Sutton et al., 2022], is elegantly captured using the options framework [Sutton et al., 1999].

However,macro-action design isoften heuristic and rarely comes with performance guarantees.

This is an interesting approach and seems novel in the context of options, although it looks to have some similarities to potential based reward shaping, e.g. (Devlin and Kudenko, 2012). The main advantages claimed for HAAR are (loosely) those of improved performance under sparse rewards and the learning of skills appropriate for transfer. These claims could be made more explicit, and that might help to justify the experimental section. The authors define advantage as: A_h(s_t h,a_t h) E[r_t h \gamma_h V_h(s_{t k} h) - V_h(s_{t} h)] The meaning of this is a little ambiguous and I would prefer this to be clarified.

General-purpose agents require fine-grained controls and rich sensory inputs to perform a wide range of tasks. However, this complexity often leads to intractable decision-making. Traditionally, agents are provided with task-specific action and observation spaces to mitigate this challenge, but this reduces autonomy. Instead, agents must be capable of building state-action spaces at the correct abstraction level from their sensorimotor experiences. We leverage the structure of a given set of temporally-extended actions to learn abstract Markov decision processes (MDPs) that operate at a higher level of temporal and state granularity. We characterize state abstractions necessary to ensure that planning with these skills, by simulating trajectories in the abstract MDP, results in policies with bounded value loss in the original MDP. We evaluate our approach in goal-based navigation environments that require continuous abstract states to plan successfully and show that abstract model learning improves the sample efficiency of planning and learning.

We introduce a framework that enables an agent to autonomously learn its own symbolic representation of a low-level, continuous environment. Propositional symbols are formalized as names for probability distributions, providing a natural means of dealing with uncertain representations and probabilistic plans. We determine the symbols that are sufficient for computing the probability with which a plan will succeed, and demonstrate the acquisition of a symbolic representation in a computer game domain.

The goal of transfer is to use knowledge obtained by solving one task to improvea robot's (or software agent's) performance in future tasks. In general, we do not expect this to work; for transfer to be feasible, there must be something in common between the source task(s) and goal task(s). The question at the core of the transfer learning enterprise is therefore: what makes two tasks related?, or more generally, how do you define a family of related tasks? Given a precise definition of how a particular family of tasks is related, we can formulate clear optimizationmethods for selecting source tasks and determining what knowledge should be imported from the source task(s), and how it should be used in the target task(s). This paper describes one model that has appeared in several different research scenarios where an agent is faced with afamily of tasks that have similar, but not identical, dynamics (or reward functions). For example, a human learning to play baseball may, over the course of their career,be exposed to several different bats, each with slightly different weights and lengths.A human who has learned to play baseball well with one bat would be expected to be able to pick up any similar bat and use it.Similarly, when learning to drive a car, one may learn in more than one car, and then be expected to be able to drive any make and model of car (within reasonablevariations) with little or no relearning. These examples are instances of exactly the kind of flexible, reliable,and sample-efficient behavior that we should be aiming to achieve in robotics applications. One way to model such a family of tasks is to posit that they are generated by asmall set of latent parameters (e.g., the length and weight of the bat, or parametersdescribing the various physical properties of the car's steering system and clutch) thatare fixed for each problem instance (e.g., for each bat, or car), but are not directlyobservable by the agent. Defining a distributionover these latent parameters results in a family of related tasks, and transferis feasible to the extent that the number of latent variables is small, the task dynamics(or reward function) vary smoothly with them, and to the extent to which they can eitherbe ignored or identified using transition data from the task.This model has appeared under several different names in the literature; we refer to it as a hidden-parameterMarkov decision process (or HIP-MDP).

Autonomous discovery of discrete symbols and rules from continuous interaction experience is a crucial building block of robot AI, but remains a challenging problem. Solving it will overcome the limitations in scalability, flexibility, and robustness of manually-designed symbols and rules, and will constitute a substantial advance towards autonomous robots that can learn and reason at abstract levels in open-ended environments. Towards this goal, we propose a novel and general method that finds action-grounded, discrete object and effect categories and builds probabilistic rules over them that can be used in complex action planning. Our robot interacts with single and multiple objects using a given action repertoire and observes the effects created in the environment. In order to form action-grounded object, effect, and relational categories, we employ a binarized bottleneck layer of a predictive, deep encoder-decoder network that takes as input the image of the scene and the action applied, and generates the resulting object displacements in the scene (action effects) in pixel coordinates. The binary latent vector represents a learned, action-driven categorization of objects. To distill the knowledge represented by the neural network into rules useful for symbolic reasoning, we train a decision tree to reproduce its decoder function. From its branches we extract probabilistic rules and represent them in PPDDL, allowing off-the-shelf planners to operate on the robot's sensorimotor experience. Our system is verified in a physics-based 3d simulation environment where a robot arm-hand system learned symbols that can be interpreted as 'rollable', 'insertable', 'larger-than' from its push and stack actions; and generated effective plans to achieve goals such as building towers from given cubes, balls, and cups using off-the-shelf probabilistic planners.

George Konidaris still remembers his disheartening introduction to robotics. "When you're a young student and you want to program a robot, the first thing that hits you is this immense disappointment at how much you can't do with that robot," he says. Most new roboticists want to program their robots to solve interesting, complex tasks -- but it turns out that just moving them through space without colliding with objects is more difficult than it sounds. Fortunately, Konidaris is hopeful that future roboticists will have a more exciting start in the field. That's because roughly four years ago, he co-founded Realtime Robotics, a startup that's solving the "motion planning problem" for robots.

E.g., its search space was shown to be compatible with symbolic Goal Recognition [Amado et al., 2018]. We achieved a new milestone in the difficult task One major drawback of the previous work was that it used of enabling agents to learn about their environment a non-descriptive, black-box neural model as the successor autonomously. Our neuro-symbolic architecture is generator. Not only that such a black-box model is incompatible trained end-to-end to produce a succinct and effective with the existing heuristic search techniques, but also, discrete state transition model from images since a neural network is able to model a very complex function, alone. Our target representation (the Planning Domain its direct translation into a compact logical formula via Definition Language) is already in a form that a rule-based transfer learning method turned out futile [Asai, off-the-shelf solvers can consume, and opens the 2019a]: The model complexity causes an exponentially large door to the rich array of modern heuristic search grounded action model that cannot be processed by the modern capabilities. We demonstrate how the sophisticated classical planners. Thus, obtaining the descriptive action innate prior we place on the learning process significantly models from the raw observations with minimal human interference reduces the complexity of the learned representation, is the next key milestone for expanding the scope of and reveals a connection to the graphtheoretic applying Automated Planning to the raw unstructured inputs.