Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics

Chen, W; Gong, X; Wu, J; Wei, Y; Shi, H; Yan, Z; Yang, Y; Wang, Z

Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics

Chen, W Gong, X Wu, J Wei, Y Shi, H Yan, Z Yang, Y

Wang, Z

Permalink

Publisher:: IEEE COMPUTER SOC
Publication Type:: Journal Article
Citation:: IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46, (2), pp. 749-763
Issue Date:: 2024-02-01

Open Access

Copyright Clearance Process

Recently Added
In Progress
Open Access

This item is open access.

Adobe PDF

Download Published versionAdobe PDF (8.92 MB)

View on publisher's site

View statistics

Full metadata record

Field	Value	Language
dc.contributor.author	Chen, W
dc.contributor.author	Gong, X
dc.contributor.author	Wu, J
dc.contributor.author	Wei, Y
dc.contributor.author	Shi, H
dc.contributor.author	Yan, Z
dc.contributor.author	Yang, Y https://orcid.org/0000-0002-0512-880X
dc.contributor.author	Wang, Z
dc.date.accessioned	2024-08-07T04:47:27Z
dc.date.available	2024-08-07T04:47:27Z
dc.date.issued	2024-02-01
dc.identifier.citation	IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46, (2), pp. 749-763
dc.identifier.issn	0162-8828
dc.identifier.issn	1939-3539
dc.identifier.uri	http://hdl.handle.net/10453/180232
dc.description.abstract	This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation. NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. Recent NAS works Mellor et al. 2021, Chen et al. 2021, Abdelfattah et al. 2021 start to explore indicators that can predict a network's performance without training. However, they either leveraged limited properties of deep networks, or the benefits of their training-free indicators were not applied to more extensive search methods. By rigorous correlation analysis, we present a unified framework to understand and accelerate NAS, by disentangling 'TEG' characteristics of searched networks - Trainability, Expressivity, Generalization - all assessed in a training-free manner. The TEG indicators could be scaled up and integrated with various NAS search methods, including both supernet and single-path NAS approaches. Extensive studies validate the effective and efficient guidance from our TEG-NAS framework, leading to both improved search accuracy and over 56% reduction in search time cost. Moreover, we visualize search trajectories on three landscapes of 'TEG' characteristics, observing that a good local minimum is easier to find on NAS-Bench-201 given its simple topology, whereas balancing 'TEG' characteristics is much harder on the DARTS space due to its complex landscape geometry.
dc.language	English
dc.publisher	IEEE COMPUTER SOC
dc.relation.ispartof	IEEE Transactions on Pattern Analysis and Machine Intelligence
dc.relation.isbasedon	10.1109/TPAMI.2023.3328347
dc.rights	info:eu-repo/semantics/openAccess
dc.subject	0801 Artificial Intelligence and Image Processing, 0806 Information Systems, 0906 Electrical and Electronic Engineering
dc.subject.classification	Artificial Intelligence & Image Processing
dc.subject.classification	4603 Computer vision and multimedia computation
dc.subject.classification	4611 Machine learning
dc.title	Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
dc.type	Journal Article
utslib.citation.volume	46
utslib.for	0801 Artificial Intelligence and Image Processing
utslib.for	0806 Information Systems
utslib.for	0906 Electrical and Electronic Engineering
pubs.organisational-group	University of Technology Sydney
pubs.organisational-group	University of Technology Sydney/Faculty of Engineering and Information Technology
utslib.copyright.status	open_access	*
dc.rights.license	This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0). To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/4.0/
dc.date.updated	2024-08-07T04:47:15Z
pubs.issue	2
pubs.publication-status	Published
pubs.volume	46
utslib.citation.issue	2

Abstract:

This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation. NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. Recent NAS works Mellor et al. 2021, Chen et al. 2021, Abdelfattah et al. 2021 start to explore indicators that can predict a network's performance without training. However, they either leveraged limited properties of deep networks, or the benefits of their training-free indicators were not applied to more extensive search methods. By rigorous correlation analysis, we present a unified framework to understand and accelerate NAS, by disentangling 'TEG' characteristics of searched networks - Trainability, Expressivity, Generalization - all assessed in a training-free manner. The TEG indicators could be scaled up and integrated with various NAS search methods, including both supernet and single-path NAS approaches. Extensive studies validate the effective and efficient guidance from our TEG-NAS framework, leading to both improved search accuracy and over 56% reduction in search time cost. Moreover, we visualize search trajectories on three landscapes of 'TEG' characteristics, observing that a good local minimum is easier to find on NAS-Bench-201 given its simple topology, whereas balancing 'TEG' characteristics is much harder on the DARTS space due to its complex landscape geometry.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/180232