EEG-Based Contrastive Learning Models For Object Perception Using Multisensory Image-Audio Stimuli
- Publisher:
- Association for Computing Machinery (ACM)
- Publication Type:
- Conference Proceeding
- Citation:
- BCIMM 2024 - Proceedings of the 1st International Workshop on Brain-Computer Interfaces BCI for Multimedia Understanding, Co-Located with: MM 2024, 2024, pp. 39-47
- Issue Date:
- 2024-10-28
Open Access
Copyright Clearance Process
- Recently Added
- In Progress
- Open Access
This item is open access.
Multimedia sources such as images and audio commonly activate human senses to perceive objects, but limited research has explored the combined effect of these stimuli on predicting semantic object perception. In this study, we compare the performance of EEG signals elicited by image and audio stimuli in classifying semantic objects, revealing that image stimuli are more discriminative than audio stimuli. Building on this, we developed a contrastive learning model that integrates image and audio stimuli, further enhancing classification performance. Our research makes several key contributions: it compares classifier performance with uni-sensory versus multisensory stimuli, demonstrates improved performance with contrastive learning models using EEG data from both image and audio stimuli, and introduces a novel method to generate positive and negative pairs for contrastive learning models using cross-sensory EEG data. These findings enhance our understanding of how humans perceive multimedia sources and highlight the potential of multisensory integration in EEG-based classification.
Please use this identifier to cite or link to this item: