{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hard-patches-mining-for-masked-image-modeling","title":"Hard Patches Mining for Masked Image Modeling","arxiv_id":"2304.05919","date":"2023-04-12","proceeding":"CVPR 2023 1","authors":["Haochen Wang","Kaiyou Song","Junsong Fan","Yuxi Wang","Jin Xie","Zhaoxiang Zhang"],"abstract":"Masked image modeling (MIM) has attracted much research attention due to its promising potential for learning scalable visual representations. In typical approaches, models usually focus on predicting specific contents of masked patches, and their performances are highly related to pre-defined mask strategies. Intuitively, this procedure can be considered as training a student (the model) on solving given problems (predict masked patches). However, we argue that the model should not only focus on solving given problems, but also stand in the shoes of a teacher to produce a more challenging problem by itself. To this end, we propose Hard Patches Mining (HPM), a brand-new framework for MIM pre-training. We observe that the reconstruction loss can naturally be the metric of the difficulty of the pre-training task. Therefore, we introduce an auxiliary loss predictor, predicting patch-wise losses first and deciding where to mask next. It adopts a relative relationship learning strategy to prevent overfitting to exact reconstruction loss values. Experiments under various settings demonstrate the effectiveness of HPM in constructing masked images. Furthermore, we empirically find that solely introducing the loss prediction objective leads to powerful representations, verifying the efficacy of the ability to be aware of where is hard to reconstruct.","url_abs":"https://arxiv.org/abs/2304.05919v1","url_pdf":"https://arxiv.org/pdf/2304.05919v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hard-patches-mining-for-masked-image-modeling","repo_url":"https://github.com/haochen-wang409/hpm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"aware","method_name":"AWARE"},{"method_slug":"mim","method_name":"MIM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2304.05919","atlas_url":"https://app.syntology.ai/?focus=2304.05919","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.05919"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haochen-wang409/hpm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Haochen-Wang409/HPM","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d88b08a86f1b802d","entry":"MaskedAutoencoderViT","repo":"Haochen-Wang409/HPM","repo_kind":"official","path":"models_mae_learn_loss.py","file_url":"https://github.com/Haochen-Wang409/HPM/blob/HEAD/models_mae_learn_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d88b08a86f1b802d"}},{"code_sha256_prefix":"39e6b23b55f376ea","entry":"build_model","repo":"haochen-wang409/hpm","repo_kind":"official","path":"models_clip.py","file_url":"https://github.com/haochen-wang409/hpm/blob/HEAD/models_clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"39e6b23b55f376ea"}},{"code_sha256_prefix":"f21bce0c57b13edf","entry":"forward_features","repo":"haochen-wang409/hpm","repo_kind":"official","path":"engine_pretrain.py","file_url":"https://github.com/haochen-wang409/hpm/blob/HEAD/engine_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f21bce0c57b13edf"}},{"code_sha256_prefix":"632d641fabe00eb8","entry":"forward_features_dino","repo":"haochen-wang409/hpm","repo_kind":"official","path":"engine_pretrain.py","file_url":"https://github.com/haochen-wang409/hpm/blob/HEAD/engine_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"632d641fabe00eb8"}},{"code_sha256_prefix":"ab5c235c1a0730f0","entry":"knn_classifier","repo":"haochen-wang409/hpm","repo_kind":"official","path":"main_knn.py","file_url":"https://github.com/haochen-wang409/hpm/blob/HEAD/main_knn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab5c235c1a0730f0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}