Dark Knowledge Balance Learning for Unbiased Scene Graph Generation

Chen, Z; Luo, Y; Shao, J; Yang, Y; Wang, C; Chen, L; Xiao, J

Dark Knowledge Balance Learning for Unbiased Scene Graph Generation

Chen, Z Luo, Y Shao, J Yang, Y

Wang, C Chen, L Xiao, J

Permalink

Publisher:: ACM
Publication Type:: Conference Proceeding
Citation:: MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 4838-4847
Issue Date:: 2023-10-26

Closed Access

	Filename	Description	Size
	3581783.3612031.pdf	Published version	5.62 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Chen, Z
dc.contributor.author	Luo, Y
dc.contributor.author	Shao, J
dc.contributor.author	Yang, Y https://orcid.org/0000-0002-0512-880X
dc.contributor.author	Wang, C
dc.contributor.author	Chen, L
dc.contributor.author	Xiao, J
dc.date	2023-10-29
dc.date.accessioned	2024-06-09T08:12:23Z
dc.date.available	2024-06-09T08:12:23Z
dc.date.issued	2023-10-26
dc.identifier.citation	MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 4838-4847
dc.identifier.isbn	979-8-4007-0108-5
dc.identifier.uri	http://hdl.handle.net/10453/179462
dc.description.abstract	One of the major obstacles that hinders the current scene graph generation (SGG) performance lies in the severe predicate annotation bias. Conventional solutions to this problem are mainly based on reweighting/resampling heuristics. Despite achieving some improvements on tail classes, these methods are prone to cause serious performance degradation of head predicates. In this paper, we propose to tackle this problem from a brand-new perspective of dark knowledge. In consideration of the unique nature of SGG that requires a large number of negative samples to be employed for predicate learning, we design to capitalize on the dark knowledge contained in negative samples for debiasing the predicate distribution. Along such vein, we propose a novel SGG method dubbed Dark Knowledge Balance Learning (DKBL). In DKBL, we first design a dark knowledge balancing loss, which helps the model learn to balance head and tail predicates while maintaining the overall performance. We further introduce a dark knowledge semantic enhancement module to better encode the semantics of predicates. DKBL is orthogonal to existing SGG methods and can be easily plugged into their training process for further improvement. Extensive experiments on VG dataset show that the proposed DKBL can consistently achieve well trade-off performance between head and tail predicates, which is significantly better than previous state-of-the-art methods. The code is available in https://github.com/chenzqing/DKBL.
dc.language	en
dc.publisher	ACM
dc.relation.ispartof	MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia
dc.relation.ispartof	ACM International Conference on Multimedia
dc.relation.isbasedon	10.1145/3581783.3612031
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	Dark Knowledge Balance Learning for Unbiased Scene Graph Generation
dc.type	Conference Proceeding
utslib.location.activity	Ottawa, Canada
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
utslib.copyright.status	closed_access	*
pubs.consider-herdc	false
dc.date.updated	2024-06-09T08:12:19Z
pubs.finish-date	2023-11-03
pubs.place-of-publication	USA
pubs.publication-status	Published
pubs.start-date	2023-10-29
dc.location	USA

Abstract:

One of the major obstacles that hinders the current scene graph generation (SGG) performance lies in the severe predicate annotation bias. Conventional solutions to this problem are mainly based on reweighting/resampling heuristics. Despite achieving some improvements on tail classes, these methods are prone to cause serious performance degradation of head predicates. In this paper, we propose to tackle this problem from a brand-new perspective of dark knowledge. In consideration of the unique nature of SGG that requires a large number of negative samples to be employed for predicate learning, we design to capitalize on the dark knowledge contained in negative samples for debiasing the predicate distribution. Along such vein, we propose a novel SGG method dubbed Dark Knowledge Balance Learning (DKBL). In DKBL, we first design a dark knowledge balancing loss, which helps the model learn to balance head and tail predicates while maintaining the overall performance. We further introduce a dark knowledge semantic enhancement module to better encode the semantics of predicates. DKBL is orthogonal to existing SGG methods and can be easily plugged into their training process for further improvement. Extensive experiments on VG dataset show that the proposed DKBL can consistently achieve well trade-off performance between head and tail predicates, which is significantly better than previous state-of-the-art methods. The code is available in https://github.com/chenzqing/DKBL.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/179462