{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-latent-masked-image-modeling-for-self","title":"Towards Latent Masked Image Modeling for Self-Supervised Visual Representation Learning","arxiv_id":"2407.15837","date":"2024-07-22","proceeding":null,"authors":["Yibing Wei","Abhinav Gupta","Pedro Morgado"],"abstract":"Masked Image Modeling (MIM) has emerged as a promising method for deriving visual representations from unlabeled image data by predicting missing pixels from masked portions of images. It excels in region-aware learning and provides strong initializations for various tasks, but struggles to capture high-level semantics without further supervised fine-tuning, likely due to the low-level nature of its pixel reconstruction objective. A promising yet unrealized framework is learning representations through masked reconstruction in latent space, combining the locality of MIM with the high-level targets. However, this approach poses significant training challenges as the reconstruction targets are learned in conjunction with the model, potentially leading to trivial or suboptimal solutions.Our study is among the first to thoroughly analyze and address the challenges of such framework, which we refer to as Latent MIM. Through a series of carefully designed experiments and extensive analysis, we identify the source of these challenges, including representation collapsing for joint online/target optimization, learning objectives, the high region correlation in latent space and decoding conditioning. By sequentially addressing these issues, we demonstrate that Latent MIM can indeed learn high-level representations while retaining the benefits of MIM models.","url_abs":"https://arxiv.org/abs/2407.15837v1","url_pdf":"https://arxiv.org/pdf/2407.15837v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-latent-masked-image-modeling-for-self","repo_url":"https://github.com/yibingwei-1/latentmim","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"mim","method_name":"MIM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.15837","atlas_url":"https://app.syntology.ai/?focus=2407.15837","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.15837"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yibingwei-1/latentmim","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/yibingwei-1/LatentMIM","reach":{"status":"ok"}}],"summary":{"ran":18,"ran_draft_wrong":2,"unverified":9},"by_repo_kind":{"official":{"samples":29,"ran":20,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":29,"samples":[{"code_sha256_prefix":"14d22cad6c3652df","entry":"AbsolutePositionEmbeds","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14d22cad6c3652df"}},{"code_sha256_prefix":"97fb93bd98cfce58","entry":"Attention","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"97fb93bd98cfce58"}},{"code_sha256_prefix":"88d3a859bbb353fa","entry":"Block","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"88d3a859bbb353fa"}},{"code_sha256_prefix":"0aa2506ff4669ad9","entry":"CrossAttention","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0aa2506ff4669ad9"}},{"code_sha256_prefix":"09c1e9e18a95d73c","entry":"CrossBlock","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09c1e9e18a95d73c"}},{"code_sha256_prefix":"94c80da22c9b99f5","entry":"CrossPositionAttentionStem","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94c80da22c9b99f5"}},{"code_sha256_prefix":"8936931f961fac53","entry":"PatchEmbed","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8936931f961fac53"}},{"code_sha256_prefix":"e6455c9712d66617","entry":"Patchify","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6455c9712d66617"}},{"code_sha256_prefix":"630c0648178f5810","entry":"Similarity","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"630c0648178f5810"}},{"code_sha256_prefix":"e287eb398f89f6ed","entry":"SinCosPEs","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e287eb398f89f6ed"}},{"code_sha256_prefix":"d4031567204db418","entry":"_build_mlp","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d4031567204db418"}},{"code_sha256_prefix":"4a3f7a34b495299c","entry":"adjust_target_momentum","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"main_lmim.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/main_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a3f7a34b495299c"}},{"code_sha256_prefix":"3dc19396537db789","entry":"all_reduce_mean","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/util/misc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3dc19396537db789"}},{"code_sha256_prefix":"1fe90f92040cf053","entry":"avg_pairwise_sim","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fe90f92040cf053"}},{"code_sha256_prefix":"84c0965ec39be82d","entry":"concat_all_gather","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"util/dist_utils.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/util/dist_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84c0965ec39be82d"}},{"code_sha256_prefix":"8cacb1e15bf1d910","entry":"concat_all_gather","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/util/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8cacb1e15bf1d910"}},{"code_sha256_prefix":"f25dd0ec378284fd","entry":"cosine_decay_schedule","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"main_lmim.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/main_lmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f25dd0ec378284fd"}},{"code_sha256_prefix":"44c6311e21193939","entry":"generate_fewshot_dataset","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"data_tools.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/data_tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44c6311e21193939"}},{"code_sha256_prefix":"aa97db87276a3a8e","entry":"read_and_split_data","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"data_tools.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/data_tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa97db87276a3a8e"}},{"code_sha256_prefix":"f71fd7e968578cce","entry":"read_splits","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"data_tools.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/data_tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f71fd7e968578cce"}},{"code_sha256_prefix":"f9294385468d37e7","entry":"CrossDecoder","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9294385468d37e7"}},{"code_sha256_prefix":"e102fd2444a9a5ae","entry":"LMIM","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e102fd2444a9a5ae"}},{"code_sha256_prefix":"9f056dcd045ee6cf","entry":"VIT","repo":"yibingwei-1/LatentMIM","repo_kind":"official","path":"models_lmim.py","file_url":"https://github.com/yibingwei-1/LatentMIM/blob/HEAD/models_lmim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f056dcd045ee6cf"}},{"code_sha256_prefix":"80500372237a4794","entry":"adjust_learning_rate","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"util/lr_sched.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/util/lr_sched.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80500372237a4794"}},{"code_sha256_prefix":"b80212c1206d00ab","entry":"build_vit","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"vits.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/vits.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b80212c1206d00ab"}},{"code_sha256_prefix":"ba1356e8ceb654d2","entry":"get_grad_norm_","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"util/misc.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/util/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba1356e8ceb654d2"}},{"code_sha256_prefix":"975cbd8043721e2c","entry":"imagenet","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"975cbd8043721e2c"}},{"code_sha256_prefix":"2479657c36a797a0","entry":"imagenet100","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2479657c36a797a0"}},{"code_sha256_prefix":"980a8e933845de91","entry":"load_dataset","repo":"yibingwei-1/latentmim","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/yibingwei-1/latentmim/blob/HEAD/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"980a8e933845de91"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}