{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/denoising-vision-transformers","title":"Denoising Vision Transformers","arxiv_id":"2401.02957","date":"2024-01-05","proceeding":null,"authors":["Jiawei Yang","Katie Z Luo","Jiefeng Li","Congyue Deng","Leonidas Guibas","Dilip Krishnan","Kilian Q Weinberger","Yonglong Tian","Yue Wang"],"abstract":"We study a crucial yet often overlooked issue inherent to Vision Transformers (ViTs): feature maps of these models exhibit grid-like artifacts, which hurt the performance of ViTs in downstream dense prediction tasks such as semantic segmentation, depth prediction, and object discovery. We trace this issue down to the positional embeddings at the input stage. To mitigate this, we propose a two-stage denoising approach, termed Denoising Vision Transformers (DVT). In the first stage, we separate the clean features from those contaminated by positional artifacts by enforcing cross-view feature consistency with neural fields on a per-image basis. This per-image optimization process extracts artifact-free features from raw ViT outputs, providing clean feature estimates for offline applications. In the second stage, we train a lightweight transformer block to predict clean features from raw ViT outputs, leveraging the derived estimates of the clean features as supervision. Our method, DVT, does not require re-training the existing pre-trained ViTs, and is immediately applicable to any Vision Transformer architecture. We evaluate our method on a variety of representative ViTs (DINO, DeiT-III, EVA02, CLIP, DINOv2, DINOv2-reg) and demonstrate that DVT consistently improves existing state-of-the-art general-purpose models in semantic and geometric tasks across multiple datasets. We hope our study will encourage a re-evaluation of ViT design, especially regarding the naive use of positional embeddings. Our code and checkpoints are publicly available.","url_abs":"https://arxiv.org/abs/2401.02957v2","url_pdf":"https://arxiv.org/pdf/2401.02957v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"denoising-vision-transformers","repo_url":"https://github.com/Jiawei-Yang/Denoising-ViT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"depth-prediction","task_name":"Depth Prediction"},{"task_slug":"object-discovery","task_name":"Object Discovery"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"mae","method_name":"MAE"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.02957","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.02957"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Jiawei-Yang/Denoising-ViT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bd3578806ecf5ac2","entry":"adjust_learning_rate","repo":"Jiawei-Yang/Denoising-ViT","repo_kind":"official","path":"utils/misc.py","file_url":"https://github.com/Jiawei-Yang/Denoising-ViT/blob/HEAD/utils/misc.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd3578806ecf5ac2"}},{"code_sha256_prefix":"e8910478d04e13ef","entry":"all_reduce_mean","repo":"Jiawei-Yang/Denoising-ViT","repo_kind":"official","path":"utils/misc.py","file_url":"https://github.com/Jiawei-Yang/Denoising-ViT/blob/HEAD/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8910478d04e13ef"}},{"code_sha256_prefix":"3366160515c6d8fc","entry":"get_pca_map","repo":"Jiawei-Yang/Denoising-ViT","repo_kind":"official","path":"utils/visualization_tools.py","file_url":"https://github.com/Jiawei-Yang/Denoising-ViT/blob/HEAD/utils/visualization_tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3366160515c6d8fc"}},{"code_sha256_prefix":"2ed1ba43a0d970df","entry":"get_scale_map","repo":"Jiawei-Yang/Denoising-ViT","repo_kind":"official","path":"utils/visualization_tools.py","file_url":"https://github.com/Jiawei-Yang/Denoising-ViT/blob/HEAD/utils/visualization_tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ed1ba43a0d970df"}},{"code_sha256_prefix":"cb7a56256f3453c8","entry":"make_patch_coordinates","repo":"Jiawei-Yang/Denoising-ViT","repo_kind":"official","path":"main_img_denoising.py","file_url":"https://github.com/Jiawei-Yang/Denoising-ViT/blob/HEAD/main_img_denoising.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb7a56256f3453c8"}},{"code_sha256_prefix":"ba1356e8ceb654d2","entry":"get_grad_norm_","repo":"Jiawei-Yang/Denoising-ViT","repo_kind":"official","path":"utils/misc.py","file_url":"https://github.com/Jiawei-Yang/Denoising-ViT/blob/HEAD/utils/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba1356e8ceb654d2"}},{"code_sha256_prefix":"8b8380ca3d0bffbf","entry":"get_robust_pca","repo":"Jiawei-Yang/Denoising-ViT","repo_kind":"official","path":"utils/visualization_tools.py","file_url":"https://github.com/Jiawei-Yang/Denoising-ViT/blob/HEAD/utils/visualization_tools.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b8380ca3d0bffbf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}