{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-to-hide-from-your-students-attention","title":"What to Hide from Your Students: Attention-Guided Masked Image Modeling","arxiv_id":"2203.12719","date":"2022-03-23","proceeding":null,"authors":["Ioannis Kakogeorgiou","Spyros Gidaris","Bill Psomas","Yannis Avrithis","Andrei Bursuc","Konstantinos Karantzalos","Nikos Komodakis"],"abstract":"Transformers and masked language modeling are quickly being adopted and explored in computer vision as vision transformers and masked image modeling (MIM). In this work, we argue that image token masking differs from token masking in text, due to the amount and correlation of tokens in an image. In particular, to generate a challenging pretext task for MIM, we advocate a shift from random masking to informed masking. We develop and exhibit this idea in the context of distillation-based MIM, where a teacher transformer encoder generates an attention map, which we use to guide masking for the student. We thus introduce a novel masking strategy, called attention-guided masking (AttMask), and we demonstrate its effectiveness over random masking for dense distillation-based MIM as well as plain distillation-based self-supervised learning on classification tokens. We confirm that AttMask accelerates the learning process and improves the performance on a variety of downstream tasks. We provide the implementation code at https://github.com/gkakogeorgiou/attmask.","url_abs":"https://arxiv.org/abs/2203.12719v2","url_pdf":"https://arxiv.org/pdf/2203.12719v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-to-hide-from-your-students-attention","repo_url":"https://github.com/gkakogeorgiou/attmask","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"masked-language-modeling","task_name":"Masked Language Modeling"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.12719","atlas_url":"https://app.syntology.ai/?focus=2203.12719","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.12719"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/gkakogeorgiou/attmask","reach":null}],"summary":{"ran_draft_wrong":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"db0a04f225664439","entry":"AttMask","repo":"gkakogeorgiou/attmask","repo_kind":"official","path":"attmask.py","file_url":"https://github.com/gkakogeorgiou/attmask/blob/HEAD/attmask.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db0a04f225664439"}},{"code_sha256_prefix":"fcd495b83f8d3bf5","entry":"attention_masking","repo":"gkakogeorgiou/attmask","repo_kind":"official","path":"attmask.py","file_url":"https://github.com/gkakogeorgiou/attmask/blob/HEAD/attmask.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fcd495b83f8d3bf5"}},{"code_sha256_prefix":"bafa48f82d43cb95","entry":"get_mask","repo":"gkakogeorgiou/attmask","repo_kind":"official","path":"attmask.py","file_url":"https://github.com/gkakogeorgiou/attmask/blob/HEAD/attmask.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bafa48f82d43cb95"}},{"code_sha256_prefix":"51d9e2eb01f5f64e","entry":"show_hints","repo":"gkakogeorgiou/attmask","repo_kind":"official","path":"attmask.py","file_url":"https://github.com/gkakogeorgiou/attmask/blob/HEAD/attmask.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51d9e2eb01f5f64e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}