Transformer-Based Fusion of Body and Context for Emotion Recognition

¹ Advanced digital enterprise modeling and information retrieval (ADMIR) laboratory, Rabat IT Center, Information retrieval and data analytics team (IRDA), ENSIAS, Mohammed V University in Rabat
² SMARTiLab Laboratory, Moroccan School of Engineering Sciences (EMSI Rabat/SMARTILAB)

^* e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.
^** e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.
^*** e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.
^**** e-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.

Published online: 3 August 2026

Abstract

This paper investigates using image-based features, combining body posture and context, to automatically detect emotions in natural settings. We focus on seven emotions: happy, sad, disgust, neutral, surprise, anger, and fear , from the NCAER dataset. We applied the Vision Transformer (ViT-B16) model to each cue. When body features are extracted using the Vision Transformer (ViT-B16) , we achieve an overall accuracy of 58.16%. Similarly, for context feature extraction, we hide the face and body of the main actor in the visual scene and then apply the Vision Transformer (ViT-B16) to the remaining context-related elements, achieving an accuracy of 55.19%. However, by fusing both body and context features, we improve the accuracy to 56.67%, surpassing the GLAMOUR_Net method.