EmoDETective: Detecting, Exploring, and Thinking Emotional Causes in Videos

Abstract

There exists an affective gap between video content and the emotions induced by video creators. Existing methods for video emotional content analysis attempt to learn emotion-related features directly or enhance discriminative models but lack emotional cause descriptions, limiting their interpretability and the model’s reasoning capabilities. In this work, we introduce EmoCause, the first large-scale video emotional dataset with multi-attribute, multi-split emotional cause descriptions. EmoCause builds upon existing datasets and is divided into 14K video splits, with over 309K emotional cause descriptions. Inspired by psychology and video prior knowledge, each video split is linked to four primary emotional cause attributes: audio, visuals, content, and shot, further divided into 12 sub-attributes, with each cause including a fact and analysis. Then, we merge all the causes into an emotional chain-of-thought for the entire video to enhance the reasoning process. Furthermore, we develop a multimodal large language model (MLLM) for video emotional content analysis, EmoDETective. EmoDETective performs training on EmoCause using progressive learning, which includes Detecting cause fact, Exploring cause analysis, and Thinking with complete reasoning. Experimental results show that our approach surpasses the existing MLLMs baseline and outperforms state-of-the-art methods, demonstrating superior emotional analysis capabilities. Ablation experiments indicate improvements from both the proposed dataset and training strategy. All data, models, and training strategies will be open-sourced to further promote the development of video emotional content analysis.