Tamara van Gog

Effects of Observing and Imitating Gestures on Word Learning and Learning from Animations

In this talk I will present research that suggests that the effectiveness of gesture observation and imitation for learning, may depend on the learning material as well as learner characteristics. For instance, studies by De Nooijer, Van Gog, Paas, and Zwaan on the effects of observing and imitating gestures on children’s learning of new verbs, showed that the effects of gesture observation and imitation depend on verb type as well as verbal abilities of the child. Findings by Post, Van Gog, Paas, and Zwaan, suggest that simultaneously or sequentially observing and imitating gestures while learning an abstract procedure (grammar rule transformation) from instructional animations, does not help learning and may even hinder learning for children with lower language abilities.

Asli Ozyurek

Neural and Cognitive Infrastructure of Multimodal Language Processing and Learning: Implications for Multimedia Technology

Language use in face-to face context is multimodal. It requires interlocuters to produce and perceive communicative messages not only using speech but also from other visual channels such as from lips, face, eye gaze and hand gestures. Furthermore deaf communicaties rely solely on visible actions to use language and communicate. The visible and in some cases iconic (e.g., drinking gesture to mean drink) nature of such visible communicative actions (as opposed to arbitrary ones as in speech) suggest that they might involve embodied procesing. In this talk I will review research showing that production and perception of such visible and multimodal signals and their learning indeed interact with sensorimotor processes (e.g., motor, action) and yet at the same time recruit brain areas involved in abstract semantic processes .
Furthermore they are designed and perceived and in orchestration with content and timing of speech and in relation to the communicative intent of the speaker and the perceived knowledge of the addressee.
Technological advances targeting multimedia tools need to take these design features of multimodal processing into account to be effective in learning.