{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/drive-segment-unsupervised-semantic","title":"Drive&Segment: Unsupervised Semantic Segmentation of Urban Scenes via Cross-modal Distillation","arxiv_id":"2203.11160","date":"2022-03-21","proceeding":null,"authors":["Antonin Vobecky","David Hurych","Oriane Siméoni","Spyros Gidaris","Andrei Bursuc","Patrick Pérez","Josef Sivic"],"abstract":"This work investigates learning pixel-wise semantic image segmentation in urban scenes without any manual annotation, just from the raw non-curated data collected by cars which, equipped with cameras and LiDAR sensors, drive around a city. Our contributions are threefold. First, we propose a novel method for cross-modal unsupervised learning of semantic image segmentation by leveraging synchronized LiDAR and image data. The key ingredient of our method is the use of an object proposal module that analyzes the LiDAR point cloud to obtain proposals for spatially consistent objects. Second, we show that these 3D object proposals can be aligned with the input images and reliably clustered into semantically meaningful pseudo-classes. Finally, we develop a cross-modal distillation approach that leverages image data partially annotated with the resulting pseudo-classes to train a transformer-based model for image semantic segmentation. We show the generalization capabilities of our method by testing on four different testing datasets (Cityscapes, Dark Zurich, Nighttime Driving and ACDC) without any finetuning, and demonstrate significant improvements compared to the current state of the art on this problem. See project webpage https://vobecant.github.io/DriveAndSegment/ for the code and more.","url_abs":"https://arxiv.org/abs/2203.11160v2","url_pdf":"https://arxiv.org/pdf/2203.11160v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"drive-segment-unsupervised-semantic","repo_url":"https://github.com/vobecant/DriveAndSegment","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"unsupervised-semantic-segmentation","task_name":"Unsupervised Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/unsupervised-semantic-segmentation-on-acdc","task":"Unsupervised Semantic Segmentation","dataset":"ACDC (Adverse Conditions Dataset with Correspondences)","model":"Segmenter ViT-S/16","rank_in_archive_order":1,"of":1,"metrics":{"mIoU":"16.7"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-on-1","task":"Unsupervised Semantic Segmentation","dataset":"Cityscapes val","model":"Segmenter ViT-S/16","rank_in_archive_order":1,"of":1,"metrics":{"mIoU":"21.8"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-on-dark","task":"Unsupervised Semantic Segmentation","dataset":"Dark Zurich","model":"Segmenter ViT-S/16","rank_in_archive_order":1,"of":1,"metrics":{"mIoU":"14.2"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-on-2","task":"Unsupervised Semantic Segmentation","dataset":"Nighttime Driving","model":"Segmenter ViT-S/16","rank_in_archive_order":1,"of":1,"metrics":{"mIoU":"18.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.11160","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.11160"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vobecant/DriveAndSegment","reach":null}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"65a1db567df04703","entry":"FPNDecoder","repo":"vobecant/DriveAndSegment","repo_kind":"official","path":"segmenter_model/fpn_picie.py","file_url":"https://github.com/vobecant/DriveAndSegment/blob/HEAD/segmenter_model/fpn_picie.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65a1db567df04703"}},{"code_sha256_prefix":"4656a057f2d1d8ce","entry":"PanopticFPN","repo":"vobecant/DriveAndSegment","repo_kind":"official","path":"segmenter_model/fpn_picie.py","file_url":"https://github.com/vobecant/DriveAndSegment/blob/HEAD/segmenter_model/fpn_picie.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4656a057f2d1d8ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}