Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings
- Publisher:
- Association for Computing Machinery (ACM)
- Publication Type:
- Conference Proceeding
- Citation:
- BCIMM 2024 - Proceedings of the 1st International Workshop on Brain-Computer Interfaces BCI for Multimedia Understanding, Co-Located with: MM 2024, 2024, pp. 19-28
- Issue Date:
- 2024-10-28
Open Access
Copyright Clearance Process
- Recently Added
- In Progress
- Open Access
This item is open access.
Decoding linguistic information from non-invasive brain signals using EEG has gained increasing research attention due to its vast applicational potential. Recently, a number of works have adopted a generative-based framework to decode electroencephalogram signals into sentences by utilizing the power generative capacity of pretrained large language models. However, this approach has several drawbacks that hinder the further development of linguistic applications for brain-computer interfaces. Specifically, the ability of the EEG encoder to learn semantic information from EEG data remains questionable, and the LLM decoder’s tendency to generate sentences based on its training memory can be hard to avoid. These issues necessitate a novel approach for converting EEG signals into sentences. In this paper, we propose a novel two-step pipeline that addresses these limitations and enhances the validity of linguistic EEG decoding research. We first confirm that word-level semantic information can be learned from EEG data recorded during natural reading by training a Conformer encoder via a masked contrastive objective for word-level classification. To achieve sentence decoding results, we employ a training-free retrieval method to retrieve sentences based on the predictions from the EEG encoder. Our evaluation results demonstrate that our EEG encoder achieves up to 55.15% top-20 classification accuracy with visualization results validating its ability to learn from unspoken EEG recordings. Subsequently, using the predicted classification results, our retrieval method attains a recall@5 of up to 55.55% for sentence-level evaluation. Despite the exploratory nature of this work, these results suggest that our method holds promise for providing more reliable solutions for converting EEG signals into text.
Please use this identifier to cite or link to this item: