Gesture based word (re)acquisition with a virtual agent in augmented reality: A preliminary study

From an evolutionary perspective, language and gesture belong together\r\nas a system, serving communication on both an abstract and a physical level. In\r\naphasia, when language is impaired, patients make use of gestures. Laboratory\r\nresearch has provided evidence that gesture can support aphasia rehabilitation, or\r\nmore specifically, anomia rehabilitation. Here, we test an anomia gesture-based\r\nrehabilitation scenario with a virtual trainer (VT) in augmented reality (AR) as a\r\ntherapy simulation. Thirty German-speaking participants were trained on 27 biand\r\nthree-syllabic words of Vimmi, an artificial language. Each Vimmi word was\r\npaired to a function word in German. The participants were divided into two\r\nGroups of 15 and 15 persons. Group A learned word pairs by observing the\r\ngestures performed by the VT and additionally imitating them. Group B learned 27\r\nword-pairs by observing the VT standing still and listening to them. Participants\r\nwere trained singularly for 3 days, alternating one day of training with one day of\r\nrest for memory consolidation. Word retention was assessed immediately after\r\neach training session by means of free and cued recall tests administered\r\nelectronically. Group A and Group B did not differ in word retention. When\r\nsubdividing participants in high and low performers, interactions showed that high\r\nperformers benefitted more from gesture-based training than low performers. The\r\ndata in this preliminary study do not speak in favour of VTs as possible tools in\r\ngesture-based AR language rehabilitation. Technology might have, in this case,\r\ndetrimental effects on word learning.