I-athlon: Towards A Multidimensional Turing Test

While the Turing test is a well-known method for evaluating machine intelligence, it has a number of drawbacks that make it problematic as a rigorous and practical test for assessing progress in general-purpose AI. For example, the Turing test is deception based, subjectively evaluated, and narrowly focused on language use. We suggest that a test would benefit from including the following requirements: focus on rational behavior, test several dimensions of intelligence, automate as much as possible, score as objectively as possible, and allow incremental progress to be measured. The approach, which we call the I-athlon, is intended to ultimately enable the community to evaluate progress towards machine intelligence in a practical and repeatable way.