Benchmarks, Test Beds, Controlled Experimentation, and the Design of Agent Architectures

The methodological underpinnings of AI are slowly changing. Benchmarks, test beds, and controlled experimentation are becoming more common. Although we are optimistic that this change can solidify the science of AI, we also recognize a set of difficult issues concerning the appropriate use of this methodology. We discuss these issues as they relate to research on agent design. We survey existing test beds for agents and argue for appropriate caution in their use. We end with a debate on the proper role of experimental methodology in the design and validation of planning agents.