Graph Attention for Automated Audio Captioning

Xiao, F; Guan, J; Zhu, Q; Wang, W

Graph Attention for Automated Audio Captioning

Xiao, F Guan, J Zhu, Q

Wang, W

Permalink

Publisher:: IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
Publication Type:: Journal Article
Citation:: IEEE Signal Processing Letters, 2023, 30, pp. 413-417
Issue Date:: 2023-01-01

Embargoed

	Filename	Description	Size
	Graph_Attention_for_Automated_Audio_Captioning.pdf	Accepted version	489.99 kB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Embargoed
Open Access

This item is currently unavailable due to the publisher's embargo.

The embargo period expires on 11 Apr 2025

Full metadata record

Field	Value	Language
dc.contributor.author	Xiao, F
dc.contributor.author	Guan, J
dc.contributor.author	Zhu, Q https://orcid.org/0000-0003-3942-0945
dc.contributor.author	Wang, W
dc.date.accessioned	2023-11-06T05:47:57Z
dc.date.available	2023-11-06T05:47:57Z
dc.date.issued	2023-01-01
dc.identifier.citation	IEEE Signal Processing Letters, 2023, 30, pp. 413-417
dc.identifier.issn	1070-9908
dc.identifier.issn	1558-2361
dc.identifier.uri	http://hdl.handle.net/10453/173062
dc.description.abstract	State-of-the-art audio captioning methods typically use the encoder-decoder structure with pretrained audio neural networks (PANNs) as encoders for feature extraction. However, the convolution operation used in PANNs is limited in capturing the long-time dependencies within an audio signal, thereby leading to potential performance degradation in audio captioning. This letter presents a novel method using graph attention (GraphAC) for encoder-decoder based audio captioning. In the encoder, a graph attention module is introduced after the PANNs to learn contextual association (i.e. the dependency among the audio features over different time frames) through an adjacency graph, and a top-k mask is used to mitigate the interference from noisy nodes. The learnt contextual association leads to a more effective feature representation with feature node aggregation. As a result, the decoder can predict important semantic information about the acoustic scene and events based on the contextual associations learned from the audio signal. Experimental results show that GraphAC outperforms the state-of-the-art methods with PANNs as the encoders, thanks to the incorporation of the graph attention module into the encoder for capturing the long-time dependencies within the audio signal.
dc.language	English
dc.publisher	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
dc.relation.ispartof	IEEE Signal Processing Letters
dc.relation.isbasedon	10.1109/LSP.2023.3266114
dc.rights	info:eu-repo/semantics/embargoedAccess
dc.rights	©2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
dc.subject	0801 Artificial Intelligence and Image Processing, 0906 Electrical and Electronic Engineering, 1005 Communications Technologies
dc.subject.classification	Networking & Telecommunications
dc.subject.classification	4006 Communications engineering
dc.subject.classification	4009 Electronics, sensors and digital hardware
dc.subject.classification	4603 Computer vision and multimedia computation
dc.title	Graph Attention for Automated Audio Captioning
dc.type	Journal Article
utslib.citation.volume	30
utslib.for	0801 Artificial Intelligence and Image Processing
utslib.for	0906 Electrical and Electronic Engineering
utslib.for	1005 Communications Technologies
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology/School of Mechanical and Mechatronic Engineering
utslib.copyright.status	embargoed	*
utslib.copyright.embargo	2025-04-11T00:00:00+1000Z
dc.date.updated	2023-11-06T05:47:56Z
pubs.publication-status	Published
pubs.volume	30

Abstract:

State-of-the-art audio captioning methods typically use the encoder-decoder structure with pretrained audio neural networks (PANNs) as encoders for feature extraction. However, the convolution operation used in PANNs is limited in capturing the long-time dependencies within an audio signal, thereby leading to potential performance degradation in audio captioning. This letter presents a novel method using graph attention (GraphAC) for encoder-decoder based audio captioning. In the encoder, a graph attention module is introduced after the PANNs to learn contextual association (i.e. the dependency among the audio features over different time frames) through an adjacency graph, and a top-k mask is used to mitigate the interference from noisy nodes. The learnt contextual association leads to a more effective feature representation with feature node aggregation. As a result, the decoder can predict important semantic information about the acoustic scene and events based on the contextual associations learned from the audio signal. Experimental results show that GraphAC outperforms the state-of-the-art methods with PANNs as the encoders, thanks to the incorporation of the graph attention module into the encoder for capturing the long-time dependencies within the audio signal.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/173062