CVLNet: Cross-view Semantic Correspondence Learning for Video-Based Camera Localization

Shi, Y; Yu, X; Wang, S; Li, H

CVLNet: Cross-view Semantic Correspondence Learning for Video-Based Camera Localization

Shi, Y Yu, X

Wang, S Li, H

Permalink

Publisher:: SPRINGER INTERNATIONAL PUBLISHING AG
Publication Type:: Chapter
Citation:: Computer Vision – ACCV 2022, 2023, 13841 LNCS, pp. 123-141
Issue Date:: 2023-01-01

Closed Access

	Filename	Description	Size
	978-3-031-26319-4_8.pdf		2.59 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Shi, Y
dc.contributor.author	Yu, X https://orcid.org/0000-0002-0269-5649
dc.contributor.author	Wang, S
dc.contributor.author	Li, H
dc.contributor.editor	Gall, J
dc.contributor.editor	Chin, TJ
dc.contributor.editor	Sato, I
dc.contributor.editor	Chellappa, R
dc.contributor.editor	Wang, L
dc.date.accessioned	2023-11-09T02:13:23Z
dc.date.available	2023-11-09T02:13:23Z
dc.date.issued	2023-01-01
dc.identifier.citation	Computer Vision – ACCV 2022, 2023, 13841 LNCS, pp. 123-141
dc.identifier.isbn	9783031263187
dc.identifier.uri	http://hdl.handle.net/10453/173260
dc.description.abstract	This paper tackles the problem of Cross-view Video-based camera Localization (CVL). The task is to localize a query camera by leveraging information from its past observations, i.e., a continuous sequence of images observed at previous time stamps, and matching them to a large overhead-view satellite image. The critical challenge of this task is to learn a powerful global feature descriptor for the sequential ground-view images while considering its domain alignment with reference satellite images. For this purpose, we introduce CVLNet, which first projects the sequential ground-view images into an overhead view by exploring the ground-and-overhead geometric correspondences and then leverages the photo consistency among the projected images to form a global representation. In this way, the cross-view domain differences are bridged. Since the reference satellite images are usually pre-cropped and regularly sampled, there is always a misalignment between the query camera location and its matching satellite image center. Motivated by this, we propose estimating the query camera’s relative displacement to a satellite image before similarity matching. In this displacement estimation process, we also consider the uncertainty of the camera location. For example, a camera is unlikely to be on top of trees. To evaluate the performance of the proposed method, we collect satellite images from Google Map for the KITTI dataset and construct a new cross-view video-based localization benchmark dataset, KITTI-CVL. Extensive experiments have demonstrated the effectiveness of video-based localization over single image-based localization and the superiority of each proposed module over other alternatives.
dc.language	en
dc.publisher	SPRINGER INTERNATIONAL PUBLISHING AG
dc.relation	http://purl.org/au-research/grants/arc/DP220100800
dc.relation	http://purl.org/au-research/grants/arc/DE230100477
dc.relation.ispartof	Computer Vision – ACCV 2022
dc.relation.ispartofseries	Lecture Notes in Computer Science
dc.relation.isbasedon	10.1007/978-3-031-26319-4_8
dc.rights	info:eu-repo/semantics/closedAccess
dc.subject.classification	Artificial Intelligence & Image Processing
dc.subject.classification	46 Information and computing sciences
dc.title	CVLNet: Cross-view Semantic Correspondence Learning for Video-Based Camera Localization
dc.type	Chapter
utslib.citation.volume	13841 LNCS
pubs.organisational-group	/University of Technology Sydney
pubs.organisational-group	/University of Technology Sydney/Faculty of Engineering and Information Technology
pubs.organisational-group	/University of Technology Sydney/Strength - AAII - Australian Artificial Intelligence Institute
utslib.copyright.status	closed_access	*
dc.date.updated	2023-11-09T02:13:22Z
pubs.publication-status	Published
pubs.volume	13841 LNCS

Abstract:

This paper tackles the problem of Cross-view Video-based camera Localization (CVL). The task is to localize a query camera by leveraging information from its past observations, i.e., a continuous sequence of images observed at previous time stamps, and matching them to a large overhead-view satellite image. The critical challenge of this task is to learn a powerful global feature descriptor for the sequential ground-view images while considering its domain alignment with reference satellite images. For this purpose, we introduce CVLNet, which first projects the sequential ground-view images into an overhead view by exploring the ground-and-overhead geometric correspondences and then leverages the photo consistency among the projected images to form a global representation. In this way, the cross-view domain differences are bridged. Since the reference satellite images are usually pre-cropped and regularly sampled, there is always a misalignment between the query camera location and its matching satellite image center. Motivated by this, we propose estimating the query camera’s relative displacement to a satellite image before similarity matching. In this displacement estimation process, we also consider the uncertainty of the camera location. For example, a camera is unlikely to be on top of trees. To evaluate the performance of the proposed method, we collect satellite images from Google Map for the KITTI dataset and construct a new cross-view video-based localization benchmark dataset, KITTI-CVL. Extensive experiments have demonstrated the effectiveness of video-based localization over single image-based localization and the superiority of each proposed module over other alternatives.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/173260