{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/token-labeling-training-a-85-5-top-1-accuracy","title":"All Tokens Matter: Token Labeling for Training Better Vision Transformers","arxiv_id":"2104.10858","date":"2021-04-22","proceeding":"NeurIPS 2021 12","authors":["Zihang Jiang","Qibin Hou","Li Yuan","Daquan Zhou","Yujun Shi","Xiaojie Jin","Anran Wang","Jiashi Feng"],"abstract":"In this paper, we present token labeling -- a new training objective for training high-performance vision transformers (ViTs). Different from the standard training objective of ViTs that computes the classification loss on an additional trainable class token, our proposed one takes advantage of all the image patch tokens to compute the training loss in a dense manner. Specifically, token labeling reformulates the image classification problem into multiple token-level recognition problems and assigns each patch token with an individual location-specific supervision generated by a machine annotator. Experiments show that token labeling can clearly and consistently improve the performance of various ViT models across a wide spectrum. For a vision transformer with 26M learnable parameters serving as an example, with token labeling, the model can achieve 84.4% Top-1 accuracy on ImageNet. The result can be further increased to 86.4% by slightly scaling the model size up to 150M, delivering the minimal-sized model among previous models (250M+) reaching 86%. We also show that token labeling can clearly improve the generalization capability of the pre-trained models on downstream tasks with dense prediction, such as semantic segmentation. Our code and all the training details will be made publicly available at https://github.com/zihangJiang/TokenLabeling.","url_abs":"https://arxiv.org/abs/2104.10858v3","url_pdf":"https://arxiv.org/pdf/2104.10858v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"token-labeling-training-a-85-5-top-1-accuracy","repo_url":"https://github.com/zihangJiang/TokenLabeling","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"token-labeling-training-a-85-5-top-1-accuracy","repo_url":"https://github.com/catalpaaa/demansia","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"token-labeling-training-a-85-5-top-1-accuracy","repo_url":"https://github.com/flytocc/TokenLabeling-paddle","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"paddle","reach":{"status":"ok"}},{"paper_slug":"token-labeling-training-a-85-5-top-1-accuracy","repo_url":"https://github.com/naver-ai/vidt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"token-labeling-training-a-85-5-top-1-accuracy","repo_url":"https://github.com/sail-sg/dualformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"token-labeling-training-a-85-5-top-1-accuracy","repo_url":"https://github.com/zhoudaquan/Refiner_ViT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"token-labeling-training-a-85-5-top-1-accuracy","repo_url":"https://github.com/PaddlePaddle/PASSL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"efficient-vits","task_name":"Efficient ViTs"},{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"lv-vit","method_name":"LV-ViT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-lv-vit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (With LV-ViT-S)","model":"Base (LV-ViT-S)","rank_in_archive_order":3,"of":19,"metrics":{"GFLOPs":"6.6","Top 1 Accuracy":"83.3"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"LV-ViT-L","rank_in_archive_order":151,"of":1060,"metrics":{"GFLOPs":"214.8","Number of params":"151M","Top 1 Accuracy":"86.4%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"LV-ViT-M","rank_in_archive_order":356,"of":1060,"metrics":{"GFLOPs":"16","Number of params":"56M","Top 1 Accuracy":"84.1%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"LV-ViT-S","rank_in_archive_order":438,"of":1060,"metrics":{"GFLOPs":"6.6","Number of params":"26M","Top 1 Accuracy":"83.3%"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"LV-ViT-L (UperNet, MS)","rank_in_archive_order":91,"of":235,"metrics":{"Params (M)":"209","Validation mIoU":"51.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.10858","atlas_url":"https://app.syntology.ai/?focus=2104.10858","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.10858"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/naver-ai/vidt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sail-sg/dualformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PASSL","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhoudaquan/Refiner_ViT","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/catalpaaa/demansia","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zihangJiang/TokenLabeling","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/flytocc/TokenLabeling-paddle","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1},"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"9ebf0e5695d4429d","entry":"Position_embedding","repo":"zhoudaquan/Refiner_ViT","repo_kind":"listed","path":"models/refined_transformer.py","file_url":"https://github.com/zhoudaquan/Refiner_ViT/blob/HEAD/models/refined_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ebf0e5695d4429d"}},{"code_sha256_prefix":"b240fa8e2e8f8ec6","entry":"get_dpr","repo":"zihangJiang/TokenLabeling","repo_kind":"official","path":"tlt/models/lvvit.py","file_url":"https://github.com/zihangJiang/TokenLabeling/blob/HEAD/tlt/models/lvvit.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b240fa8e2e8f8ec6"}},{"code_sha256_prefix":"c8df8fc3cbc72444","entry":"get_points_single","repo":"zhoudaquan/Refiner_ViT","repo_kind":"listed","path":"models/refined_transformer.py","file_url":"https://github.com/zhoudaquan/Refiner_ViT/blob/HEAD/models/refined_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8df8fc3cbc72444"}},{"code_sha256_prefix":"219807603e18aac6","entry":"get_block","repo":"zihangJiang/TokenLabeling","repo_kind":"official","path":"tlt/models/lvvit.py","file_url":"https://github.com/zihangJiang/TokenLabeling/blob/HEAD/tlt/models/lvvit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"219807603e18aac6"}},{"code_sha256_prefix":"d34a085d71b8f4b5","entry":"rand_bbox","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d34a085d71b8f4b5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}