{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/siamese-image-modeling-for-self-supervised","title":"Siamese Image Modeling for Self-Supervised Vision Representation Learning","arxiv_id":"2206.01204","date":"2022-06-02","proceeding":"CVPR 2023 1","authors":["Chenxin Tao","Xizhou Zhu","Weijie Su","Gao Huang","Bin Li","Jie zhou","Yu Qiao","Xiaogang Wang","Jifeng Dai"],"abstract":"Self-supervised learning (SSL) has delivered superior performance on a variety of downstream vision tasks. Two main-stream SSL frameworks have been proposed, i.e., Instance Discrimination (ID) and Masked Image Modeling (MIM). ID pulls together representations from different views of the same image, while avoiding feature collapse. It lacks spatial sensitivity, which requires modeling the local structure within each image. On the other hand, MIM reconstructs the original content given a masked image. It instead does not have good semantic alignment, which requires projecting semantically similar views into nearby representations. To address this dilemma, we observe that (1) semantic alignment can be achieved by matching different image views with strong augmentations; (2) spatial sensitivity can benefit from predicting dense representations with masked images. Driven by these analysis, we propose Siamese Image Modeling (SiameseIM), which predicts the dense representations of an augmented view, based on another masked view from the same image but with different augmentations. SiameseIM uses a Siamese network with two branches. The online branch encodes the first view, and predicts the second view's representation according to the relative positions between these two views. The target branch produces the target by encoding the second view. SiameseIM can surpass both ID and MIM on a wide range of downstream tasks, including ImageNet finetuning and linear probing, COCO and LVIS detection, and ADE20k semantic segmentation. The improvement is more significant in few-shot, long-tail and robustness-concerned scenarios. Code shall be released at https://github.com/fundamentalvision/Siamese-Image-Modeling.","url_abs":"https://arxiv.org/abs/2206.01204v3","url_pdf":"https://arxiv.org/pdf/2206.01204v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"siamese-image-modeling-for-self-supervised","repo_url":"https://github.com/fundamentalvision/siamese-image-modeling","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"siamese-image-modeling-for-self-supervised","repo_url":"https://github.com/fundamentalvision/unigrad","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"sensitivity","task_name":"Sensitivity"}],"methods":[{"method_slug":"mim","method_name":"MIM"},{"method_slug":"siamese-network","method_name":"Siamese Network"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2206.01204","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.01204"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fundamentalvision/unigrad","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fundamentalvision/siamese-image-modeling","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"unverified":5},"by_repo_kind":{"listed":{"samples":7,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"131a82fd65128218","entry":"accuracy","repo":"fundamentalvision/unigrad","repo_kind":"listed","path":"lineval.py","file_url":"https://github.com/fundamentalvision/unigrad/blob/HEAD/lineval.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"131a82fd65128218"}},{"code_sha256_prefix":"63ce1d5edf8fb0a0","entry":"validate","repo":"fundamentalvision/unigrad","repo_kind":"listed","path":"lineval.py","file_url":"https://github.com/fundamentalvision/unigrad/blob/HEAD/lineval.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"63ce1d5edf8fb0a0"}},{"code_sha256_prefix":"28f62e0c183cc93f","entry":"build_dataloader","repo":"fundamentalvision/unigrad","repo_kind":"listed","path":"project/dataloader.py","file_url":"https://github.com/fundamentalvision/unigrad/blob/HEAD/project/dataloader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"28f62e0c183cc93f"}},{"code_sha256_prefix":"014a0e97a9d25469","entry":"build_model","repo":"fundamentalvision/unigrad","repo_kind":"listed","path":"project/model.py","file_url":"https://github.com/fundamentalvision/unigrad/blob/HEAD/project/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"014a0e97a9d25469"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"fundamentalvision/unigrad","repo_kind":"listed","path":"project/utils.py","file_url":"https://github.com/fundamentalvision/unigrad/blob/HEAD/project/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"3637b63ede9883b8","entry":"get_logger","repo":"fundamentalvision/unigrad","repo_kind":"listed","path":"project/utils.py","file_url":"https://github.com/fundamentalvision/unigrad/blob/HEAD/project/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3637b63ede9883b8"}},{"code_sha256_prefix":"ed80ac83a60f2121","entry":"init_environ","repo":"fundamentalvision/unigrad","repo_kind":"listed","path":"project/utils.py","file_url":"https://github.com/fundamentalvision/unigrad/blob/HEAD/project/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed80ac83a60f2121"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}