The interaction of observation and inference in a formal representation system