Task Communication Through Natural Language and Graphics

With increases in the complexity of information that must be communicated either by or to computers comes a corresponding need to find ways to communicate that information simply and effectively. It makes little sense to force the burden of communication on a single medium, restricted to just one of spoken or written text, gestures, diagrams, or graphical animation, when in many situations information is only communicated effectively through combinations of media. In response to requests for directions, respondents often choose to provide both a sketch map (for visual indications of relative distance, spatial relationships, etc.) as well as verbal guidance as to landmarks to attend to, obstacles to watch out for, opportunities to take, etc. Instructors training a subject in a new task often choose to present the task in at least two ways: they demonstrate what motions the trainee is supposed to carry out, using direct training, film or graphic media, and they convey what intentional actions those motions are meant to represent, through naturallanguage text or speech. Graphic media (diagrams and animation) can provide a way of visualizing significant patterns in situations (cf. the current interest in Scientific Visualization), while natural-language text (either spoken or written) can provide needed information on what the patterns may mean, why they may have developed, or what may be done to deal with them. Naturallanguage narration is necessary to convey the meaning and significance of such visualizations.)