{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/augmentation-free-dense-contrastive-knowledge","title":"Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation","arxiv_id":"2312.04168","date":"2023-12-07","proceeding":null,"authors":["Jiawei Fan","Chao Li","Xiaolong Liu","Meina Song","Anbang Yao"],"abstract":"In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less attention is paid to semantic segmentation. Existing methods heavily rely on data augmentation and memory buffer, which entail high computational resource demands when applying them to handle semantic segmentation that requires to preserve high-resolution feature maps for making dense pixel-wise predictions. In order to address this problem, we present Augmentation-free Dense Contrastive Knowledge Distillation (Af-DCD), a new contrastive distillation learning paradigm to train compact and accurate deep neural networks for semantic segmentation applications. Af-DCD leverages a masked feature mimicking strategy, and formulates a novel contrastive learning loss via taking advantage of tactful feature partitions across both channel and spatial dimensions, allowing to effectively transfer dense and structured local knowledge learnt by the teacher model to a target student model while maintaining training efficiency. Extensive experiments on five mainstream benchmarks with various teacher-student network pairs demonstrate the effectiveness of our approach. For instance, the DeepLabV3-Res18|DeepLabV3-MBV2 model trained by Af-DCD reaches 77.03%|76.38% mIOU on Cityscapes dataset when choosing DeepLabV3-Res101 as the teacher, setting new performance records. Besides that, Af-DCD achieves an absolute mIOU improvement of 3.26%|3.04%|2.75%|2.30%|1.42% compared with individually trained counterpart on Cityscapes|Pascal VOC|Camvid|ADE20K|COCO-Stuff-164K. Code is available at https://github.com/OSVAI/Af-DCD","url_abs":"https://arxiv.org/abs/2312.04168v1","url_pdf":"https://arxiv.org/pdf/2312.04168v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"augmentation-free-dense-contrastive-knowledge","repo_url":"https://github.com/osvai/af-dcd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.04168","atlas_url":"https://app.syntology.ai/?focus=2312.04168","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.04168"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/OSVAI/Af-DCD","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/osvai/af-dcd","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":7},"by_repo_kind":{"official":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"78c90947fe81ee89","entry":"EnhancedProjector","repo":"osvai/af-dcd","repo_kind":"official","path":"losses/ocmgd.py","file_url":"https://github.com/osvai/af-dcd/blob/HEAD/losses/ocmgd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"78c90947fe81ee89"}},{"code_sha256_prefix":"f959df91d5bb2cc7","entry":"MiT_B0","repo":"OSVAI/Af-DCD","repo_kind":"official","path":"models/segformer.py","file_url":"https://github.com/OSVAI/Af-DCD/blob/HEAD/models/segformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f959df91d5bb2cc7"}},{"code_sha256_prefix":"9d35034a0e342c34","entry":"MiT_B1","repo":"OSVAI/Af-DCD","repo_kind":"official","path":"models/segformer.py","file_url":"https://github.com/OSVAI/Af-DCD/blob/HEAD/models/segformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d35034a0e342c34"}},{"code_sha256_prefix":"b1f1880f9690d69a","entry":"MiT_B2","repo":"OSVAI/Af-DCD","repo_kind":"official","path":"models/segformer.py","file_url":"https://github.com/OSVAI/Af-DCD/blob/HEAD/models/segformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1f1880f9690d69a"}},{"code_sha256_prefix":"1a104eb32e82351a","entry":"OmniContrastiveFeatureLoss","repo":"osvai/af-dcd","repo_kind":"official","path":"losses/ocmgd.py","file_url":"https://github.com/osvai/af-dcd/blob/HEAD/losses/ocmgd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1a104eb32e82351a"}},{"code_sha256_prefix":"102279aea24ef5b8","entry":"get_deeplabv3_mobile","repo":"OSVAI/Af-DCD","repo_kind":"official","path":"models/deeplabv3_mobile.py","file_url":"https://github.com/OSVAI/Af-DCD/blob/HEAD/models/deeplabv3_mobile.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"102279aea24ef5b8"}},{"code_sha256_prefix":"bbc73935847f5b2a","entry":"get_psp_mobile","repo":"OSVAI/Af-DCD","repo_kind":"official","path":"models/psp_mobile.py","file_url":"https://github.com/OSVAI/Af-DCD/blob/HEAD/models/psp_mobile.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbc73935847f5b2a"}},{"code_sha256_prefix":"b5094ada5f8b0e27","entry":"get_segmentation_model","repo":"OSVAI/Af-DCD","repo_kind":"official","path":"models/model_zoo.py","file_url":"https://github.com/OSVAI/Af-DCD/blob/HEAD/models/model_zoo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5094ada5f8b0e27"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}