Affective AI and Sentiment Analysis play critical roles in designing safe and effective human
computer interactions. It can be found in applications ranging from tutor chatbots to the physical
robots used in eldercare. There is a growing concern, however, that increasingly capable AI will be
able to manipulate, persuade, and otherwise compromise human autonomy. Rapid progress in AI is
providing a constant stream of new and more capable LLM/LMMs that can better understand
nuanced, complex, and interrelated sentiments across different modalities including text, vision, and
speech. This paper introduces MultiSentimentArcs, a novel methodology that combines sentiment
analysis, time series transformations, narrative studies, and leading open-source AI models to analyze
two main modalities of sentiment expression in long-form video narratives. Although we use
Hollywood films to demonstrate the technique, it can generalize to any long-form multimodal
narrative like those found on social media (text, images), in video conversations (text, image, voice),
or in medical settings (text, data, voice, image). To the best of our knowledge, MultiSentimentArcs is
the first framework to integrate multimodal Affective AI that enables human-in-the-loop exploration,
analysis, and explanation of long-form narratives. To support the democratization of AI, all
components are open-source and can run on mid-range consumer gaming laptops. This research can
significantly advance the field of Digital Humanities by giving non-AI experts access to directly
engage in human-in-the-loop research around Affective AI and human-AI alignment. Code, results,
and non-copyrighted data will be available at https://github.com/jon-chun/multisentimentarcs.
Royal Wedding (1951) Video Sentiment Arcs
Royal Wedding (1951) Video Sentiment Arcs
Royal Wedding (1951) Transcript Sentiment Arcs
Royal Wedding (1951) Video Sentiment KDE Distribution
Royal Wedding (1951) Transcripts Sentiment KDE Distribution
NOTE:
This repo is in the process of being cleaned-up and reorganized. Please be patient
