{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/global-local-self-distillation-for-visual","title":"Global-Local Self-Distillation for Visual Representation Learning","arxiv_id":"2207.14676","date":"2022-07-29","proceeding":null,"authors":["Tim Lebailly","Tinne Tuytelaars"],"abstract":"The downstream accuracy of self-supervised methods is tightly linked to the proxy task solved during training and the quality of the gradients extracted from it. Richer and more meaningful gradients updates are key to allow self-supervised methods to learn better and in a more efficient manner. In a typical self-distillation framework, the representation of two augmented images are enforced to be coherent at the global level. Nonetheless, incorporating local cues in the proxy task can be beneficial and improve the model accuracy on downstream tasks. This leads to a dual objective in which, on the one hand, coherence between global-representations is enforced and on the other, coherence between local-representations is enforced. Unfortunately, an exact correspondence mapping between two sets of local-representations does not exist making the task of matching local-representations from one augmentation to another non-trivial. We propose to leverage the spatial information in the input images to obtain geometric matchings and compare this geometric approach against previous methods based on similarity matchings. Our study shows that not only 1) geometric matchings perform better than similarity based matchings in low-data regimes but also 2) that similarity based matchings are highly hurtful in low-data regimes compared to the vanilla baseline without local self-distillation. The code is available at https://github.com/tileb1/global-local-self-distillation.","url_abs":"https://arxiv.org/abs/2207.14676v2","url_pdf":"https://arxiv.org/pdf/2207.14676v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"global-local-self-distillation-for-visual","repo_url":"https://github.com/tileb1/global-local-self-distillation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2207.14676","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.14676"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tileb1/global-local-self-distillation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":2,"ran_honours":1,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":"tileb1/global-local-self-distillation","repo_kind":"official","path":"layers/performer.py","file_url":"https://github.com/tileb1/global-local-self-distillation/blob/HEAD/layers/performer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"tileb1/global-local-self-distillation","repo_kind":"official","path":"layers/performer.py","file_url":"https://github.com/tileb1/global-local-self-distillation/blob/HEAD/layers/performer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"3444251df88e6169","entry":"init_","repo":"tileb1/global-local-self-distillation","repo_kind":"official","path":"layers/linformer.py","file_url":"https://github.com/tileb1/global-local-self-distillation/blob/HEAD/layers/linformer.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3444251df88e6169"}},{"code_sha256_prefix":"757aee54843af070","entry":"softmax_kernel","repo":"tileb1/global-local-self-distillation","repo_kind":"official","path":"layers/performer.py","file_url":"https://github.com/tileb1/global-local-self-distillation/blob/HEAD/layers/performer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"757aee54843af070"}},{"code_sha256_prefix":"f2a753560fea8427","entry":"get_model_name","repo":"tileb1/global-local-self-distillation","repo_kind":"official","path":"config/models.py","file_url":"https://github.com/tileb1/global-local-self-distillation/blob/HEAD/config/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2a753560fea8427"}},{"code_sha256_prefix":"89d15b99aa3a0e0b","entry":"get_norm","repo":"tileb1/global-local-self-distillation","repo_kind":"official","path":"layers/batch_norm.py","file_url":"https://github.com/tileb1/global-local-self-distillation/blob/HEAD/layers/batch_norm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89d15b99aa3a0e0b"}},{"code_sha256_prefix":"2758a65d287143b7","entry":"swish","repo":"tileb1/global-local-self-distillation","repo_kind":"official","path":"layers/attention.py","file_url":"https://github.com/tileb1/global-local-self-distillation/blob/HEAD/layers/attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2758a65d287143b7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}