While in the past much of the empirical RL However, analyzing the nature of research has focused on tabular or linear function approximation those environments is often overlooked. In particular, case (Dietterich, 1998; McGovern & Barto, 2001; we still do not have agreeable ways to Konidaris & Barto, 2009), the impressive successes of recent measure the difficulty or solvability of a task, years (and anticipation of domains ripe for subsequent given that each has fundamentally different actions, successes) has spurred the creation of non-tabular benchmarks observations, dynamics, rewards, and can - i.e., continuous control and/or continuous observation be tackled with diverse RL algorithms. In this - in which neural network function approximators are work, we propose policy information capacity effectively a prerequisite (Bellemare et al., 2013; Brockman (PIC) - the mutual information between policy parameters et al., 2016; Tassa et al., 2018). Accordingly, empirical RL and episodic return - and policy-optimal research is presently heavily focused on the use of neural information capacity (POIC) - between policy network function approximators, spurring new algorithmic parameters and episodic optimality - as two developments in both model-free (Mnih et al., 2015; Schulman environment-agnostic, algorithm-agnostic quantitative et al., 2015; Lillicrap et al., 2016; Gu et al., 2016b; metrics for task difficulty. Evaluating our 2017; Haarnoja et al., 2018) and model-based (Chua et al., metrics across toy environments as well as continuous 2018; Janner et al., 2019; Hafner et al., 2020a) RL. control benchmark tasks from OpenAI Gym and DeepMind Control Suite, we empirically Despite the impressive progress of RL algorithms, the analysis demonstrate that these information-theoretic of the RL environments has been difficult and stagnant, metrics have higher correlations with normalized precisely due to the complexity of modern benchmarks and task solvability scores than a variety of alternatives.