Pose-robust personalized facial expression recognition through unsupervised multi-source domain adaptation

Abstract

Pose-robust personalized facial expression recognition is rather challenging, as facial expressions are subject-related and pose-dependent. Multi-source domain adaptation tries to leverage knowledge from multiple source domains to boost the performance of the target domain. For pose-robust personalized facial expression recognition, the images of the source domain are from multiple sources since the images are under different poses. Thus, in this paper, we propose a novel unsupervised multi-source domain adaptation framework for pose-robust personalized facial expression recognition. The proposed framework consists of five components: a source encoder, a target encoder, an expression classifier, a view discriminator, and a domain discriminator. The source encoder and target encoder learn facial representations from facial images in the training and testing sets, respectively. The expression classifier recognizes expressions from the learned representations. The view discriminator classifies poses. The domain discriminator distinguishes the learned representations of the source domain from those of the target domain. The source encoder works cooperatively with the expression classifier and adversarially with the view discriminator. The target encoder aims to learn domain robust representations and fool the domain discriminator, while the domain discriminator tries to distinguish between the source and target domains. Through adversarial learning, the distribution of the learned representations from the source domain converges to that from the target domain. Thus, the feature representation extracted by the target encoder is pose-invariant and target subject-specific. Experimental results demonstrate the superiority of the proposed method compared to related works.