{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vilt-vision-and-language-transformer-without","title":"ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision","arxiv_id":"2102.03334","date":"2021-02-05","proceeding":null,"authors":["Wonjae Kim","Bokyung Son","Ildoo Kim"],"abstract":"Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.","url_abs":"https://arxiv.org/abs/2102.03334v2","url_pdf":"https://arxiv.org/pdf/2102.03334v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vilt-vision-and-language-transformer-without","repo_url":"https://github.com/dandelin/vilt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"vilt-vision-and-language-transformer-without","repo_url":"https://github.com/glamor-usc/climb","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"vilt-vision-and-language-transformer-without","repo_url":"https://github.com/guilk/vlc","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"vilt-vision-and-language-transformer-without","repo_url":"https://github.com/huggingface/transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"vilt-vision-and-language-transformer-without","repo_url":"https://github.com/wglab/gestaltmml","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"vilt-vision-and-language-transformer-without","repo_url":"https://github.com/wglab/gestaltmml-gestaltgpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"multimodal-intent-recognition","task_name":"Multimodal Intent Recognition"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"},{"task_slug":"zero-shot-cross-modal-retrieval","task_name":"Zero-Shot Cross-Modal Retrieval"},{"task_slug":"multimodal-interaction","task_name":"multimodal interaction"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vilt","method_name":"ViLT"}],"datasets_introduced":[],"methods_introduced":[{"slug":"vilt","name":"ViLT","full_name":"Vision-and-Language Transformer"}],"results":[{"leaderboard":"/sota/cross-modal-retrieval-on-coco-2014","task":"Cross-Modal Retrieval","dataset":"COCO 2014","model":"ViLT-B/32","rank_in_archive_order":27,"of":36,"metrics":{"Image-to-text R@1":"61.5","Image-to-text R@10":"92.7","Image-to-text R@5":"86.3","Text-to-image R@1":"42.7","Text-to-image R@10":"83.1","Text-to-image R@5":"72.9"},"uses_additional_data":true},{"leaderboard":"/sota/cross-modal-retrieval-on-flickr30k","task":"Cross-Modal Retrieval","dataset":"Flickr30k","model":"ViLT-B/32","rank_in_archive_order":13,"of":27,"metrics":{"Image-to-text R@1":"83.5","Image-to-text R@10":"98.6","Image-to-text R@5":"96.7","Text-to-image R@1":"64.4","Text-to-image R@10":"93.8","Text-to-image R@5":"88.7"},"uses_additional_data":true},{"leaderboard":"/sota/image-retrieval-on-photochat","task":"Image Retrieval","dataset":"PhotoChat","model":"ViLT","rank_in_archive_order":3,"of":5,"metrics":{"R1":"11.5","R@10":"25.6","R@5":"33.8","Sum(R@1,5,10)":"71.0"},"uses_additional_data":false},{"leaderboard":"/sota/multimodal-intent-recognition-on-mmdialog","task":"Multimodal Intent Recognition","dataset":"MMDialog","model":"ViLT","rank_in_archive_order":4,"of":4,"metrics":{"F1":"55.8"},"uses_additional_data":false},{"leaderboard":"/sota/multimodal-intent-recognition-on-photochat","task":"Multimodal Intent Recognition","dataset":"PhotoChat","model":"ViLT","rank_in_archive_order":5,"of":6,"metrics":{"F1":"52.4","Precision":"55.4","Recall":"58.9"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-test-dev","task":"Visual Question Answering (VQA)","dataset":"VQA v2 test-dev","model":"ViLT-B/32","rank_in_archive_order":25,"of":56,"metrics":{"Accuracy":"71.26"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-nlvr2-dev","task":"Visual Reasoning","dataset":"NLVR2 Dev","model":"ViLT-B/32","rank_in_archive_order":13,"of":15,"metrics":{"Accuracy":"75.7"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-nlvr2-test","task":"Visual Reasoning","dataset":"NLVR2 Test","model":"ViLT-B/32","rank_in_archive_order":14,"of":14,"metrics":{"Accuracy":"76.13"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-cross-modal-retrieval-on-coco-2014","task":"Zero-Shot Cross-Modal Retrieval","dataset":"COCO 2014","model":"ViLT-B/32","rank_in_archive_order":16,"of":18,"metrics":{"Image-to-text R@1":"56.5","Image-to-text R@10":"89.6","Image-to-text R@5":"82.6","Text-to-image R@1":"40.4","Text-to-image R@10":"81.1","Text-to-image R@5":"70"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-cross-modal-retrieval-on-flickr30k","task":"Zero-Shot Cross-Modal Retrieval","dataset":"Flickr30k","model":"ViLT-B/32","rank_in_archive_order":19,"of":22,"metrics":{"Image-to-text R@1":"73.2","Image-to-text R@10":"96.5","Image-to-text R@5":"93.6","Text-to-image R@1":"55","Text-to-image R@10":"89.8","Text-to-image R@5":"82.5"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2102.03334","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.03334"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/dandelin/vilt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/guilk/vlc","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/glamor-usc/climb","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wglab/gestaltmml","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wglab/gestaltmml-gestaltgpt","reach":{"status":"unanswered"}}],"summary":{"ran":1,"unverified":3},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":3,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"f5e8d3006a5a1a6f","entry":"EncoderWrapper","repo":"glamor-usc/climb","repo_kind":"listed","path":"src/modeling/vilt.py","file_url":"https://github.com/glamor-usc/climb/blob/HEAD/src/modeling/vilt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5e8d3006a5a1a6f"}},{"code_sha256_prefix":"57128e00df804ac7","entry":"VLCTransformer","repo":"guilk/vlc","repo_kind":"listed","path":"vlc/modules/vlc_module.py","file_url":"https://github.com/guilk/vlc/blob/HEAD/vlc/modules/vlc_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57128e00df804ac7"}},{"code_sha256_prefix":"dab2a98f28bf818f","entry":"ViLTransformerSS","repo":"dandelin/vilt","repo_kind":"official","path":"vilt/modules/vilt_module.py","file_url":"https://github.com/dandelin/vilt/blob/HEAD/vilt/modules/vilt_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dab2a98f28bf818f"}},{"code_sha256_prefix":"9e90629b7bccbd2a","entry":"ViltEncoderWrapper","repo":"glamor-usc/climb","repo_kind":"listed","path":"src/modeling/vilt.py","file_url":"https://github.com/glamor-usc/climb/blob/HEAD/src/modeling/vilt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e90629b7bccbd2a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}