{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/not-all-images-are-worth-16x16-words-dynamic","title":"Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition","arxiv_id":"2105.15075","date":"2021-05-31","proceeding":"NeurIPS 2021 12","authors":["Yulin Wang","Rui Huang","Shiji Song","Zeyi Huang","Gao Huang"],"abstract":"Vision Transformers (ViT) have achieved remarkable success in large-scale image recognition. They split every 2D image into a fixed number of patches, each of which is treated as a token. Generally, representing an image with more tokens would lead to higher prediction accuracy, while it also results in drastically increased computational cost. To achieve a decent trade-off between accuracy and speed, the number of tokens is empirically set to 16x16 or 14x14. In this paper, we argue that every image has its own characteristics, and ideally the token number should be conditioned on each individual input. In fact, we have observed that there exist a considerable number of \"easy\" images which can be accurately predicted with a mere number of 4x4 tokens, while only a small fraction of \"hard\" ones need a finer representation. Inspired by this phenomenon, we propose a Dynamic Transformer to automatically configure a proper number of tokens for each input image. This is achieved by cascading multiple Transformers with increasing numbers of tokens, which are sequentially activated in an adaptive fashion at test time, i.e., the inference is terminated once a sufficiently confident prediction is produced. We further design efficient feature reuse and relationship reuse mechanisms across different components of the Dynamic Transformer to reduce redundant computations. Extensive empirical results on ImageNet, CIFAR-10, and CIFAR-100 demonstrate that our method significantly outperforms the competitive baselines in terms of both theoretical computational efficiency and practical inference speed. Code and pre-trained models (based on PyTorch and MindSpore) are available at https://github.com/blackfeather-wang/Dynamic-Vision-Transformer and https://github.com/blackfeather-wang/Dynamic-Vision-Transformer-MindSpore.","url_abs":"https://arxiv.org/abs/2105.15075v2","url_pdf":"https://arxiv.org/pdf/2105.15075v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"not-all-images-are-worth-16x16-words-dynamic","repo_url":"https://github.com/blackfeather-wang/Dynamic-Vision-Transformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"not-all-images-are-worth-16x16-words-dynamic","repo_url":"https://github.com/blackfeather-wang/dynamic-vision-transformer-mindspore","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"mindspore","reach":{"status":"ok"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"image-classification","task_name":"Image Classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"DVT (T2T-ViT-24)","rank_in_archive_order":38,"of":265,"metrics":{"Percentage correct":"98.53"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"DVT (T2T-ViT-24)","rank_in_archive_order":28,"of":211,"metrics":{"Percentage correct":"89.63"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DVT (T2T-ViT-12)","rank_in_archive_order":701,"of":1060,"metrics":{"GFLOPs":"1.7","Top 1 Accuracy":"80.43%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DVT (T2T-ViT-10)","rank_in_archive_order":743,"of":1060,"metrics":{"GFLOPs":"0.7","Top 1 Accuracy":"79.74%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DVT (T2T-ViT-7)","rank_in_archive_order":830,"of":1060,"metrics":{"GFLOPs":"0.6","Top 1 Accuracy":"78.48%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2105.15075","atlas_url":"https://app.syntology.ai/?focus=2105.15075","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.15075"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/blackfeather-wang/Dynamic-Vision-Transformer-MindSpore","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/blackfeather-wang/Dynamic-Vision-Transformer","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/blackfeather-wang/dynamic-vision-transformer-mindspore","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"8f49cc1ffec8b4b7","entry":"dynamic_evaluate","repo":"blackfeather-wang/Dynamic-Vision-Transformer","repo_kind":"official","path":"inference.py","file_url":"https://github.com/blackfeather-wang/Dynamic-Vision-Transformer/blob/HEAD/inference.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f49cc1ffec8b4b7"}},{"code_sha256_prefix":"972a0b6e71a3501c","entry":"dynamic_find_threshold","repo":"blackfeather-wang/Dynamic-Vision-Transformer","repo_kind":"official","path":"inference.py","file_url":"https://github.com/blackfeather-wang/Dynamic-Vision-Transformer/blob/HEAD/inference.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"972a0b6e71a3501c"}},{"code_sha256_prefix":"1399a5ed1c04887e","entry":"generate_logits","repo":"blackfeather-wang/dynamic-vision-transformer","repo_kind":"official","path":"inference.py","file_url":"https://github.com/blackfeather-wang/dynamic-vision-transformer/blob/HEAD/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1399a5ed1c04887e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}