Benchmarking Intelligent Service Robots through Scientific Competitions: The RoboCup@Home Approach

The dynamical and uncertain environments of domestic service robots, which include humans, require rethinking of the benchmarking principles for testing these robots. In RoboCup@Home, statistical procedures are used to track and steer the progress of domestic service robots since 2006. This paper explains the procedures and shows outcomes of these international benchmarking efforts. Although aspects such as shopping in a supermarket receive a fair amount of attention in the robotics community, the authors think that a recently started test is the most important outcome of RoboCup@Home, namely the benchmarking of robot cognition.