Gesture based word (re)acquisition with a virtual agent in augmented reality: A preliminary study

From an evolutionary perspective, language and gesture belong together as a system, serving communication on both an abstract and a physical level. In aphasia, when language is impaired, patients make use of gestures. Laboratory research has provided evidence that gesture can support aphasia rehabilitation, or more specifically, anomia rehabilitation. Here, we test an anomia gesture-based rehabilitation scenario with a virtual trainer (VT) in augmented reality (AR) as a therapy simulation. Thirty German-speaking participants were trained on 27 biand three-syllabic words of Vimmi, an artificial language. Each Vimmi word was paired to a function word in German. The participants were divided into two Groups of 15 and 15 persons. Group A learned word pairs by observing the gestures performed by the VT and additionally imitating them. Group B learned 27 word-pairs by observing the VT standing still and listening to them. Participants were trained singularly for 3 days, alternating one day of training with one day of rest for memory consolidation. Word retention was assessed immediately after each training session by means of free and cued recall tests administered electronically. Group A and Group B did not differ in word retention. When subdividing participants in high and low performers, interactions showed that high performers benefitted more from gesture-based training than low performers. The data in this preliminary study do not speak in favour of VTs as possible tools in gesture-based AR language rehabilitation. Technology might have, in this case, detrimental effects on word learning.