Since collecting paired visible and thermal facial images is often difficult, requiring paired data during training prevents the usage of the many available unpaired visible and thermal images, and thus may degenerate the learning effect of the visible facial expression classifier.
To address this, we propose an unpaired adversarial facial expression recognition method. We tackle the unbalanced quantity of visible and thermal images by utilizing thermal images as privileged information. We introduce adversarial learning on the feature-level and label-level spaces to cope with unpaired training data. Finally, we add a decoder network to preserve the inherent visible features.