Analysis of the human multimodal emotional signals to an interactive computer

The present study tries to examine what kind of emotional responses human subjects express while interacting with an artificial agent if they believe that this agent is able to understand their emotional states. For this purpose different kind of computer games were programmed to elicit specific emotional appraisals. 30 subjects were asked to use the computer where an avatar provided a simulated intelligent feedback (experimental condition), or it only guided the subject across the different tasks (control). Multimodal synchronized data were captured: physiological data, posture, facial expressions, vocal behavior. All observational data were codified frame by frame using The Observer 5.0. THEME Software was used for the detection of recurrent patterns. This is an inprogress study: the paper includes initial results of the analysis of non-verbal communicative signals.