Mixture of Attention Variants for Modal Fusion in Multi-Modal Sentiment Analysis

doi:10.3390/bdcc8020014

科研成果详情

题名	Mixture of Attention Variants for Modal Fusion in Multi-Modal Sentiment Analysis
作者	He，Chao 1,2; Zhang，Xinghua 3; Song，Dongqing 1; Shen，Yingshan 2; Mao，Chengjie 1; Wen，Huosheng 4; Zhu ，Dingju 4; Cai，Lihua 2,4
发表日期	2024-02-01
发表期刊	Big Data and Cognitive Computing
卷号	8 期号:2
摘要	With the popularization of better network access and the penetration of personal smartphones in today’s world, the explosion of multi-modal data, particularly opinionated video messages, has created urgent demands and immense opportunities for Multi-Modal Sentiment Analysis (MSA). Deep learning with the attention mechanism has served as the foundation technique for most state-of-the-art MSA models due to its ability to learn complex inter- and intra-relationships among different modalities embedded in video messages, both temporally and spatially. However, modal fusion is still a major challenge due to the vast feature space created by the interactions among different data modalities. To address the modal fusion challenge, we propose an MSA algorithm based on deep learning and the attention mechanism, namely the Mixture of Attention Variants for Modal Fusion (MAVMF). The MAVMF algorithm includes a two-stage process: in stage one, self-attention is applied to effectively extract image and text features, and the dependency relationships in the context of video discourse are captured by a bidirectional gated recurrent neural module; in stage two, four multi-modal attention variants are leveraged to learn the emotional contributions of important features from different modalities. Our proposed approach is end-to-end and has been shown to achieve a superior performance to the state-of-the-art algorithms when tested with two largest public datasets, CMU-MOSI and CMU-MOSEI.
关键词	attention mechanism deep learning feature fusion multi-modality sentiment analysis
DOI	10.3390/bdcc8020014
URL	查看来源
语种	英语English
Scopus入藏号	2-s2.0-85185565571
引用统计
文献类型	期刊论文
条目标识符	https://repository.uic.edu.cn/handle/39GCC9TT/11634
专题	个人在本单位外知识产出
通讯作者	Cai，Lihua
作者单位	1.School of Computer Science,South China Normal University,Guangzhou,510631,China 2.Aberdeen Institute of Data Science and Artificial Intelligence,South China Normal University,Guangzhou,528225,China 3.International United College,South China Normal University,Guangzhou,528225,China 4.School of Software,South China Normal University,Guangzhou,528225,China
推荐引用方式 GB/T 7714	He，Chao,Zhang，Xinghua,Song，Dongqinget al. Mixture of Attention Variants for Modal Fusion in Multi-Modal Sentiment Analysis[J]. Big Data and Cognitive Computing, 2024, 8(2).
APA	He，Chao., Zhang，Xinghua., Song，Dongqing., Shen，Yingshan., Mao，Chengjie., .. & Cai，Lihua. (2024). Mixture of Attention Variants for Modal Fusion in Multi-Modal Sentiment Analysis. Big Data and Cognitive Computing, 8(2).
MLA	He，Chao,et al."Mixture of Attention Variants for Modal Fusion in Multi-Modal Sentiment Analysis". Big Data and Cognitive Computing 8.2(2024).