Multimodal interaction
Designs interaction with systems that understand and respond through several channels at once, such as voice, gestures, touch or gaze, which can be combined or alternated.
What problem it addresses
Every channel fails somewhere. Voice doesn’t work in noise, a screen doesn’t work while driving and gestures don’t work with busy hands.
Principles
Each channel for what it does best
Pointing says where and speech says what, as in “put that there.”
Always another way
If one channel fails or can’t be used, the same action can be done through another.
Let people choose
Combining channels isn’t forced; each person uses the one they prefer at each moment.
Applications in design
Use it like this
Decide which channel suits each situation of use and always offer an alternative through another channel.
Avoid
Relying on a single channel, such as voice, in noisy places or where speaking isn’t appropriate.
Don’t fall for it
Check that the experience works with a single channel, for people who can’t see, hear or move their hands.
Connections
Authors and works
- 1980
Richard A. Bolt
In his “Put-that-there” system, a person moved shapes on a large screen by saying “put that there” and pointing with their arm: the voice gave the command and the gesture, the place.
- 1999
Sharon Oviatt
Reviewed ten widespread but mistaken ideas about these systems and argued that, well designed, they let people express themselves better and more naturally.
Sources
Original sources
- Bolt, R. A. (1980). “Put-that-there”: Voice and gesture at the graphics interface. In Proceedings of SIGGRAPH '80 (pp. 262–270). ACM. View source: Bolt, R. A. (1980)
- Oviatt, S. (1999). Ten myths of multimodal interaction. Communications of the ACM, 42(11), 74–81. View source: Oviatt, S. (1999)
Last reviewed: