Unifying Causal Inference and Reinforcement Learning using Higher-Order Category Theory

Causal inference (Pearl, 2009a; Imbens and Rubin, 2015; Spirtes et al., 2000) and predictive state representations (PSRs) (Singh et al., 2004) in reinforcement learning (Sutton and Barto, 1998), whose roots go back to earlier work on subspace identification in linear systems (Van Overschee and De Moor, 1996) and even earlier work on algebraic theories of context-free languages Chomsky and Schützenberger (1963) and algebraic automata theory (Give'on and Arbib, 1968), both involve structure discovery of a latent variable model through interventions. The use of superficially dissimilar representations - directed acyclic graphs (DAGs) (Pearl, 1989), hybrid undirected and directed graphs (Lauritzen and Richardson, 2002) and hyperedge graphs (Forré and Mooij, 2017; Evans, 2018) in causal inference, versus Hankel matrix and Hilbert space embeddings of dynamical systems - have long obscured their deeper connections. Structure discovery in causal inference and PSRs both involve the determination of a latent structure, which is directional at lower orders, but homotopy equivalences at higher orders induce symmetries. In particular, causal inference involves determining a structure, such as a DAG that encodes direct causal effects between a pair of objects, but multiple DAG models are equivalent because of symmetries induced by conditional independences (Dawid, 2001; Studený et al., 2010a) and correlations induced by latent unobservable confounders that are only revealed over higher-order simplices (e.g., DAGs over n 3 vertices). PSRs represent "hidden state" in dynamical systems by constructing a series of tests,