Inverse Reinforcement Learning via Nonparametric Spatio-Temporal Subgoal Modeling

Advances in the eld of inverse reinforcement learning (IRL) have led to sophisticated inference frameworks that relax the original modeling assumption of observing an agent behavior that re ects only a single intention. Instead of learning a global behavioral model, recent IRL methods divide the demonstration data into parts, to account for the fact that di erent trajectories may correspond to di erent intentions, e.g., because they were generated by di erent domain experts. In this work, we go one step further: using the intuitive concept of subgoals, we build upon the premise that even a single trajectory can be explained more e ciently locally within a certain context than globally, enabling a more compact representation of the observed behavior. Based on this assumption, we build an implicit intentional model of the agents goals to forecast its behavior in unobserved situations. The result is an integrated Bayesian prediction framework that signi cantly outperforms existing IRL solutions and provides smooth policy estimates consistent with the experts plan. Most notably, our framework naturally handles situations where the intentions of the agent change over time and classical IRL algorithms fail. In addition, due to its probabilistic nature, the model can be straightforwardly applied in active learning scenarios to guide the demonstration process of the expert.