Benchmarks, Test Beds, Controlled Experimentation, and the Design of Agent Architectures

Benchmarks, test beds, and controlled experimentation are becoming more common. We discuss these issues as they relate to research on agent design. We survey existing test beds for agents and argue for appropriate caution in their use. We end with a debate on the proper role of experimental methodology in the design and validation of planning agents.