Human Pose Estimation with Shape Aware Loss
Although the mean square error (mse) of heatmap is an intuitive loss for heatmap-based human pose estimation, the joints localization …
Human Pose Estimation with Shape Aware Loss
Pose-Invariant Facial Expression Recognition
Pose-invariant facial expression recognition is quite challenging due to variations in facial appearance and self-occlusion caused by …
PDF DOI
Pose-Invariant Facial Expression Recognition
Capturing Emotion Distribution for Multimedia Emotion Tagging
To address the statistical similarity between the predicted emotion labels and ground-truth emotion labels without any assumptions, we propose a novel emotion tagging approach through adversarial learning. Specifically, the proposed emotion tagging approach consists of an emotional tag classifier C and a discriminator D.
Cite DOI
Capturing Emotion Distribution for Multimedia Emotion Tagging
Attentive One-Dimensional Heatmap Regression for Facial Landmark Detection and Tracking
The main contributions of the method are three folds. First, we are the first that propose to predict 1D heatmaps on the 𝑥 and 𝑦 axes instead of using 2D heatmaps to locate landmarks and successfully alleviate the quantization error with a fully boosted output resolution. Second, we propose a co-attention module to capture the joint coordinate distribution on the two axes. Third, based on the proposed heatmap regression method, we design a facial landmark detector and tracker which achieve state-of-the-art performance.
PDF Cite DOI
Attentive One-Dimensional Heatmap Regression for Facial Landmark Detection and Tracking
Exploiting Multi-Emotion Relations at Feature and Label Levels for Emotion Tagging
To address the shortcomings in emotion tagging, we consider applying emotion relationship patterns at both feature and label levels. We propose a novel emotion tagging framework, that makes full use of the emotion relationship patterns in local and global distribution.
PDF Cite DOI
Exploiting Multi-Emotion Relations at Feature and Label Levels for Emotion Tagging
Exploiting Self-Supervised and Semi-Supervised Learning for Facial Landmark Tracking with Unlabeled Data
We propose a new semi-supervised learning strategy which trains the tracker by regression tasks from the consistency constraints on the long facial sequence instead of two adjacent frames, such that the long-term dependencies existed in a facial sequence are captured. The proposed semisupervised learning strategy does not require any extra labels. Thus, large scale unlabeled data can be exploited for training.
PDF Cite DOI
Exploiting Self-Supervised and Semi-Supervised Learning for Facial Landmark Tracking with Unlabeled Data
Learning from Macro-expression: a Micro-expression Recognition Framework
In order to address problems in micro-expression recognition, we propose a micro-expression recognition framework that leverages macroexpression as guidance. Since subjects in macro-expression and micro-expression databases are different, Expression-Identity Disentangle Network (EIDNet) is introduced as feature extractor to disentangle expression-related features for expression samples.
PDF Cite DOI
Learning from Macro-expression: a Micro-expression Recognition Framework
Occluded Facial Expression Recognition with Step-Wise Assistance from Unpaired Non-Occluded Images
To tackle the challenges in occluded facial expression recognition, we propose a step-wise learning strategy including two types of complementary adversarial learning. In this way, the occluded classifier can learn effective information from large-scale unpaired non-occluded facial images.
PDF Cite DOI
Occluded Facial Expression Recognition with Step-Wise Assistance from Unpaired Non-Occluded Images
Unpaired Multimodal Facial Expression Recognition
Since collecting paired visible and thermal facial images is often difficult, requiring paired data during training prevents the usage of the many available unpaired visible and thermal images, and thus may degenerate the learning effect of the visible facial expression classifier. To address this, we propose an unpaired adversarial facial expression recognition method. We tackle the unbalanced quantity of visible and thermal images by utilizing thermal images as privileged information. We introduce adversarial learning on the feature-level and label-level spaces to cope with unpaired training data. Finally, we add a decoder network to preserve the inherent visible features.
PDF Cite DOI
Unpaired Multimodal Facial Expression Recognition
Capturing Spatial and Temporal Patterns for Facial Landmark Tracking through Adversarial Learning
To address the inconsistency between explicit forms of joint label distribution and the ground truth facial landmark distribution, we propose an adversarial learning framework to close the joint distribution inherent in predicted and ground truth facial landmarks.
PDF Cite DOI
Capturing Spatial and Temporal Patterns for Facial Landmark Tracking through Adversarial Learning
Dual Semi-Supervised Learning for Facial Action Unit Recognition
Instead of minimizing the distance of two joint distributions directly, which requires the estimation of the marginal distribution of the input, the proposed approach uses an adversarial strategy to exploit the probabilistic duality, thus avoiding the estimation of marginal distribution.
PDF Cite DOI
Dual Semi-Supervised Learning for Facial Action Unit Recognition
Identity- and Pose-Robust Facial Expression Recognition through Adversarial Feature Learning
Previous facial expression recognition methods either focus on pose variations or identity bias; there is no work that considers both at the same time. To this end, we propose a novel feature representation method that uses adversarial learning to overcome the challenges of both pose variations and identity bias.
PDF Cite DOI
Identity- and Pose-Robust Facial Expression Recognition through Adversarial Feature Learning
Image Aesthetic Assessment Assisted by Attributes through Adversarial Learning
In this paper, we propose a novel attributes enhanced image aesthetic assessment, where the attributes are used as privileged information.
PDF Cite DOI
Image Aesthetic Assessment Assisted by Attributes through Adversarial Learning
Integrating Facial Images, Speeches and Time for Empathy Prediction
In this paper, we propose a multi-modal deep network to predict the empathy of the listener during the conversation between two people. First, we use a bottleneck residual network proposed by to learn visual representation from facial images, and adopt fully connected network to extract audio features from the listener’s speech. Second, we propose to use the current time stage as a temporal feature, and fuse it with the learned visual and audio representations. Neural network regression is used to predict the empathy level. We further select the representative subset training data to train the proposed multi-modal deep network. Experimental results on the One-Minute Empathy Prediction dataset demonstrate the effectiveness of the proposed method.
PDF Cite DOI
Integrating Facial Images, Speeches and Time for Empathy Prediction
Occluded Facial Expression Recognition Enhanced through Privileged Information
we propose using non-occluded facial images as privileged information to assist the learning process of the occluded view. Specifically, two deep neural networks are first trained from occluded and non-occluded images respectively.
PDF Cite DOI
Occluded Facial Expression Recognition Enhanced through Privileged Information
Facial Action Unit Recognition Augmented by Their Dependencies
We propose employing the latent regression Bayesian network to effectively capture the high-order and global dependencies among AUs.
PDF Cite DOI
Facial Action Unit Recognition Augmented by Their Dependencies
Facial Expression Recognition Enhanced by Thermal Images through Adversarial Learning
In this paper, we propose a novel facial expression recognition method enhanced by thermal images. Our method leverages thermal images to construct better visible feature representation and classifiers during training through adversarial learning and similarity constraints. Specifically, we learn two deep neural networks for expression classification from visible and thermal images.
PDF Cite DOI
Facial Expression Recognition Enhanced by Thermal Images through Adversarial Learning
Personalized Multiple Facial Action Unit Recognition through Generative Adversarial Recognition Network
we propose a novel generative adversarial recognition network (GARN) for personalized AU recognition without any assumptions. Specifically, the proposed GARN consists of a generator, a discriminator, and a classifier.
PDF Cite DOI
Personalized Multiple Facial Action Unit Recognition through Generative Adversarial Recognition Network
Weakly Supervised Facial Action Unit Recognition Through Adversarial Training
We propose a novel weakly supervised AU recognition method to learn AU classifiers with only expression labels. Specifically, we notice that there exist domain knowledge about expressions and AUs that can be represented as prior probabilities. We generate pseudo AU data for each expression; for AU classifiers’ training, we propose an RAN model, which consists of a recognition model and a discrimination mode trained simultaneously by leveraging an adversarial process, to make the distribution of the recognized AU close to the distribution of the pseudo AU data. Furthermore, we extend the proposed method to semi-supervised learning with partially AU-annotated images.
PDF Cite DOI
Weakly Supervised Facial Action Unit Recognition Through Adversarial Training
A Multimodal Deep Regression Bayesian Network for Affective Video Content Analyses
In this paper, we propose a new multimodal learning method, multimodal deep regression Bayesian network (MMDRBN), to construct the high-level joint representation of visual and audio modalities for emotion tagging. Then the MMDRBN is transformed into an inference network by minimizing the KL-divergence. After that, the inference network is used to predict discrete or continuous affective scores from video content.
PDF Cite DOI
A Multimodal Deep Regression Bayesian Network for Affective Video Content Analyses
Capturing Dependencies among Labels and Features for Multiple Emotion Tagging of Multimedia Data
To the best of our knowledge, this paper is the first work to assign multiple emotions to multimedia data by exploring the emotional relationships at both feature and label levels. By learning the shared features with a multi-task RBM classifier and modeling the dependencies among emotion labels with a hierarchy RBM model, the proposed approaches can exploit both top-down and bottom-up relations among emotions independently and dependently to improve multiple emotions tagging for multimedia.
PDF Cite
Capturing Dependencies among Labels and Features for Multiple Emotion Tagging of Multimedia Data
Capturing Spatial and Temporal Patterns for Distinguishing between Posed and Spontaneous Expressions
In this paper, we introduce a novel dynamic model, termed as interval temporal restricted Boltzmann machine(IT-RBM), to jointly capture global spatial patterns and complex temporal patterns embedded in posed expressions and spontaneous expressions respectively for distinguishing posed and spontaneous expressions. The proposed IT-RBM is a three-layer hierarchical probabilistic graphical model.
PDF Cite DOI
Capturing Spatial and Temporal Patterns for Distinguishing between Posed and Spontaneous Expressions
Deep Facial Action Unit Recognition from Partially Labeled Data
Inspired by the observations that AUs are samples of the underlying AU label distributions, we propose a deep facial action unit recognition approach learning from partially AU-labeled training data through incorporating such spatial regular patterns of AU labels presented in ground-truth AU labels into the learning process of AU classifiers from a large-scale facial images without AU annotations.
PDF Cite DOI
Differentiating Between Posed and Spontaneous Expressions with Latent Regression Bayesian Network
In this paper, we propose employing the LRBN to effectively capture the high-order and global dependencies among facial geometric features
PDF Cite
Differentiating Between Posed and Spontaneous Expressions with Latent Regression Bayesian Network
Exploring Domain Knowledge for Affective Video Content Analyses
Most current works employ discriminative features and efficient classifiers for affective video content analyses, without explicitly exploring and leveraging domain knowledge for affective video content analyses. Therefore, in this paper, we propose a novel method to analyze affective video content through exploring domain knowledge. Both audio elements and visual elements are used by film makers to communicate emotions to audience. As a primary study to explore film grammar for affective video content analyses, this paper takes visual elements as an example to demonstrate the feasibility of the proposed affective video content analyses method enhanced through exploring domain knowledge.
PDF Cite DOI
Facial Expression Intensity Estimation Using Ordinal Information
Our contributions include the following aspects. First,we propose a regression approach for expression intensity estimation which exploits both ordinal relationship among different frames within an expression sequence and absolute intensity labels if available. Second, we introduce a unified max-margin learning framework to simultaneously exploit the two sources of information. An efficient algorithm to solve the optimization problem is developed. Third, our method can generalize to different learning settings depend-ing on the availability of expression intensity annotations.
PDF Cite DOI
Facial Expression Intensity Estimation Using Ordinal Information
Facial Expression Recognition with Deep two-view Support Vector Machine
Although their constructed representation reflects thermal infrared images’ supplementary role for visible images, it has no direct relationship to target expression labels. Furthermore, the hand-craft visible and thermal features may not thoroughly reflect the expression patterns embedded in images. Therefore, in this paper, we propose a new deep two-view approach to learn features from both visible and thermal images and leverage the commonality among visible and thermal images for expression recognition.
PDF Cite DOI
Facial Expression Recognition with Deep two-view Support Vector Machine