{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revisiting-the-distillation-of-image","title":"Three Pillars improving Vision Foundation Model Distillation for Lidar","arxiv_id":"2310.17504","date":"2023-10-26","proceeding":"CVPR 2024 1","authors":["Gilles Puy","Spyros Gidaris","Alexandre Boulch","Oriane Siméoni","Corentin Sautier","Patrick Pérez","Andrei Bursuc","Renaud Marlet"],"abstract":"Self-supervised image backbones can be used to address complex 2D tasks (e.g., semantic segmentation, object discovery) very efficiently and with little or no downstream supervision. Ideally, 3D backbones for lidar should be able to inherit these properties after distillation of these powerful 2D features. The most recent methods for image-to-lidar distillation on autonomous driving data show promising results, obtained thanks to distillation methods that keep improving. Yet, we still notice a large performance gap when measuring the quality of distilled and fully supervised features by linear probing. In this work, instead of focusing only on the distillation method, we study the effect of three pillars for distillation: the 3D backbone, the pretrained 2D backbones, and the pretraining dataset. In particular, thanks to our scalable distillation method named ScaLR, we show that scaling the 2D and 3D backbones and pretraining on diverse datasets leads to a substantial improvement of the feature quality. This allows us to significantly reduce the gap between the quality of distilled and fully-supervised 3D features, and to improve the robustness of the pretrained backbones to domain gaps and perturbations.","url_abs":"https://arxiv.org/abs/2310.17504v2","url_pdf":"https://arxiv.org/pdf/2310.17504v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revisiting-the-distillation-of-image","repo_url":"https://github.com/valeoai/scalr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"object-discovery","task_name":"Object Discovery"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.17504","atlas_url":"https://app.syntology.ai/?focus=2310.17504","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.17504"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/valeoai/scalr","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":4,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"f1389405b7719f00","entry":"get_layer_id_for_waffleiron","repo":"valeoai/scalr","repo_kind":"official","path":"finetune.py","file_url":"https://github.com/valeoai/scalr/blob/HEAD/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f1389405b7719f00"}},{"code_sha256_prefix":"8c3d171b49409c40","entry":"load_model_config","repo":"valeoai/scalr","repo_kind":"official","path":"finetune.py","file_url":"https://github.com/valeoai/scalr/blob/HEAD/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8c3d171b49409c40"}},{"code_sha256_prefix":"e7fcb6d9ac9deaf4","entry":"named_apply","repo":"valeoai/scalr","repo_kind":"official","path":"models/dinov2_vision_transformer.py","file_url":"https://github.com/valeoai/scalr/blob/HEAD/models/dinov2_vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7fcb6d9ac9deaf4"}},{"code_sha256_prefix":"0df1faebd15a15fa","entry":"param_groups_lrd","repo":"valeoai/scalr","repo_kind":"official","path":"finetune.py","file_url":"https://github.com/valeoai/scalr/blob/HEAD/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0df1faebd15a15fa"}},{"code_sha256_prefix":"5246673a20ebda4e","entry":"zero_pad","repo":"valeoai/scalr","repo_kind":"official","path":"datasets/pc_dataset.py","file_url":"https://github.com/valeoai/scalr/blob/HEAD/datasets/pc_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5246673a20ebda4e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}