Many Ways to Think.

Thinking, collaborating, and communicating involve interactions among the body, mind, language, and physical things. Bodily actions like pointing and nodding express intention and refer to the world. Sequences of gestures can represent environments, systems, or ideas more directly than words, as do sketches, maps, graphs, and diagrams. Creating or observing gestures and graphics can change thought for both viewers and makers. These interactions are often internal, invisible, and not easily broken into words, allowing shared and individual meanings to emerge and shift in real time. Such features challenge current multimodal AI.

Study at a glance

Abstract

Thinking, collaborating, and communication happen through interactions of the body, the mind, language, and things in the world. Actions of the body actively express intention, thought, and emotion. Points of the fingers and nods of the head can refer to things in the surrounding world. A string of interrelated gestures can represent an environment, a complex system, an arrangement of ideas, a sequence of actions. Those gestures use marks and actions in space to represent thought more directly than words-as do graphics like sketches, maps, graphs, diagrams, and pictures. Seeing or making gestures or graphics can change thought, both in those who view them and in those who make them. Interactions with gestures and graphics and the surrounding world are often internal, invisible, and unique, not in words, and not easily decomposed. Through those interactions, shared and individual meanings emerge and change in real time. These features of human thought present challenges to current multi-modal AI.