April 28, 2017

Affective posture and body movement as modality for human-computer interaction

Nadia Berthouze
Body expressions are a powerful means for technology designers to enhance people's wellbeing and social interaction.

In contrast to facial and vocal expressions, affective body movement has received relatively little attention from the affective-computing community. One reason for this is the sheer complexity and variability of postural language. However, various studies have shown how body movement can support cognitive processes, regulate emotions and mediate affective and social communication. It is, therefore, a very important communication channel that technology should exploit to improve the user experience. We are researching body movement as a way to measure and affect (or change) the quality of the user experience. We also aim at creating technology that can exploit these aspects of body movement to offer a more positive experience. We are currently addressing these questions in relation to computer games and the health-care environment. Computer games are becoming a means of choice for tackling larger issues where emotions play a very important role in creating a positive experience, e.g., in education, health and social life. In the health environment, emotion is recognized as one of the most critical aspects affecting a patient's recovery.

Contrary to what was previously thought, body posture is an important channel for affective communication. It allows discrimination between the intensity of an emotional experience and its valence. Studies in psychophysics and neuroscience have shown the importance of body configuration over body movement in recognizing emotions. Therefore, we carried out a series of studies to understand if recognition of emotional expressions from postures could be grounded into the low-level features describing posture configuration.1 Using postures acquired by a motion-capture system (see Figure 1) and statistical analysis of their configurations, we showed that people can recognize emotions expressed by faceless computer characters that reproduced the captured postures at well above chance level. We also showed that computational models based on low-level descriptions (e.g., angles between body segments) could reach recognition levels similar to those of human raters, for both acted postures and nonacted subtle expressions.2

Bodily expressions are both an affective communication channel and a powerful means to modulate a person's affective state.3 We carried out a set of experiments4 to understand how body movements imposed and afforded by new game controllers affect the quality of the player experience. We argued that technology can be designed so that body posture and movement are used as priming mechanisms to regulate people's emotional states. Our results show that when controllers do not require or afford natural body movements, we observe a complete lack of movement other than those movements necessary to facilitate game control. On the other hand, when controllers require and afford body movements that are natural to the game scenario, we observe emergence of movements that are related to enjoyment and social interaction, even though they may interfere with game performance.

Example of nonacted affective postures.

Finally, by combining these two faces of bodily expression, we have been building a system that incrementally learns to recognize affective states. We tested it on acted5 and nonacted postures2 in a computer-game context (Nintendo® Wii), where expressions can be more subtle and mixed. Our system shows very high performance comparable to that of facial-expression recognition. We are also applying our method to automatically discriminate between different communicative roles of body movement in chronic-pain patients to capture the physical reaction to pain, the affective experience related to pain and the search for empathy and attention of solicitous others. Within our Emo&Pain project,6 we are developing the system to work as a virtual coach to support and motivate chronic-pain patients during physical-exercise sessions.

In summary, our studies have shown that body expressions are a powerful means for designers of technology to enhance people's wellbeing and social interaction. However, more research is necessary to improve our understanding of the relationship between experience and movement characteristics (e.g., smoothness) and the factors (e.g., personality) that may facilitate or inhibit this relationship. This represents our next steps in this field.


Nadia Berthouze
University College London

Nadia Berthouze is a senior lecturer. She teaches and is engaged in research in the area of affective computing. The aim of her research is to create systems that can sense a user's affective state and use that information to personalise the interaction process.

  1. A. Kleinsmith, R. De Silva and N. Bianchi-Berthouze, Cross-cultural differences in recognizing affect from body posture, Interact. Comput. 18 (6), pp. 1371-1389, 2006.

  2. A. Kleinsmith and N. Bianchi-Berthouze, Modelling non-acted affective posture in a video game scenario, Proc. Int'l Conf. Kansei Eng. Emotion Res., pp. 1964-1974, 2010.

  3. J. Chandler and N. Schwarz, How extending your middle finger affects your perception of others: learned movements influence concept accessibility, J. Exp. Soc. Psychol. 45 (1), pp. 123-128, 2009.

  4. M. Pasch, N. Bianchi-Berthouze, B. van Dijk and A. Nijholt, Movement-based sports video games: investigating motivation and gaming experience, Entertain. Comput. 9 (2), pp. 169-180, 2009.

  5. N. Bianchi-Berthouze and A. Kleinsmith, A categorical approach to affective gesture recognition, Connect. Sci. 15 (4), pp. 259-269, 2003.

  6. Pain rehabilitation: E/Motion-based automated coaching (Emo&Pain) Eng. Phys. Sci. Res. Council grant EP/H007083/1. 2010–2014, http://www.uclic.ucl.ac.uk/people/n.berthouze/EPain/index.html Accessed 3 September 2010.

DOI:  10.2417/2201009.003256