{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/droppos-pre-training-vision-transformers-by-1","title":"DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions","arxiv_id":"2309.03576","date":"2023-09-07","proceeding":"NeurIPS 2023 11","authors":["Haochen Wang","Junsong Fan","Yuxi Wang","Kaiyou Song","Tong Wang","Zhaoxiang Zhang"],"abstract":"As it is empirically observed that Vision Transformers (ViTs) are quite insensitive to the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident. To address this, we present DropPos, a novel pretext task designed to reconstruct Dropped Positions. The formulation of DropPos is simple: we first drop a large random subset of positional embeddings and then the model classifies the actual position for each non-overlapping patch among all possible positions solely based on their visual appearance. To avoid trivial solutions, we increase the difficulty of this task by keeping only a subset of patches visible. Additionally, considering there may be different patches with similar visual appearances, we propose position smoothing and attentive reconstruction strategies to relax this classification problem, since it is not necessary to reconstruct their exact positions in these cases. Empirical evaluations of DropPos show strong capabilities. DropPos outperforms supervised pre-training and achieves competitive results compared with state-of-the-art self-supervised alternatives on a wide range of downstream benchmarks. This suggests that explicitly encouraging spatial reasoning abilities, as DropPos does, indeed contributes to the improved location awareness of ViTs. The code is publicly available at https://github.com/Haochen-Wang409/DropPos.","url_abs":"https://arxiv.org/abs/2309.03576v2","url_pdf":"https://arxiv.org/pdf/2309.03576v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"droppos-pre-training-vision-transformers-by-1","repo_url":"https://github.com/haochen-wang409/droppos","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"Position"},{"task_slug":"spatial-reasoning","task_name":"Spatial Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.03576","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.03576"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Haochen-Wang409/DropPos","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haochen-wang409/droppos","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"01d32b851b217f4d","entry":"build_dataset","repo":"Haochen-Wang409/DropPos","repo_kind":"official","path":"util/datasets.py","file_url":"https://github.com/Haochen-Wang409/DropPos/blob/HEAD/util/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01d32b851b217f4d"}},{"code_sha256_prefix":"b1528b5a179ebd3f","entry":"build_transform","repo":"Haochen-Wang409/DropPos","repo_kind":"official","path":"util/datasets.py","file_url":"https://github.com/Haochen-Wang409/DropPos/blob/HEAD/util/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1528b5a179ebd3f"}},{"code_sha256_prefix":"89edf5372214b57d","entry":"get_layer_id_for_vit","repo":"Haochen-Wang409/DropPos","repo_kind":"official","path":"util/lr_decay.py","file_url":"https://github.com/Haochen-Wang409/DropPos/blob/HEAD/util/lr_decay.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"89edf5372214b57d"}},{"code_sha256_prefix":"85f8b25dc390a596","entry":"param_groups_lrd","repo":"Haochen-Wang409/DropPos","repo_kind":"official","path":"util/lr_decay.py","file_url":"https://github.com/Haochen-Wang409/DropPos/blob/HEAD/util/lr_decay.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"85f8b25dc390a596"}},{"code_sha256_prefix":"387567cd37ab060e","entry":"DropPositionMaskedAutoEncoderViT","repo":"haochen-wang409/droppos","repo_kind":"official","path":"models_DropPos_mae.py","file_url":"https://github.com/haochen-wang409/droppos/blob/HEAD/models_DropPos_mae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"387567cd37ab060e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}