{"url":"/method/vatt","slug":"vatt","name":"VATT","full_name":"VATT","full_name_withheld":false,"description_markdown":"**Video-Audio-Text Transformer**, or **VATT**, is a framework for learning multimodal representations from unlabeled data using [convolution](https://paperswithcode.com/method/convolution)-free [Transformer](https://paperswithcode.com/method/transformer) architectures. Specifically, it takes raw signals as inputs and extracts multidimensional representations that are rich enough to benefit a variety of downstream tasks. VATT borrows the exact architecture from [BERT](https://paperswithcode.com/method/bert) and [ViT](https://paperswithcode.com/method/vision-transformer) except the layer of tokenization and linear projection reserved for each modality separately. The design follows the same spirit as ViT that makes the minimal changes to the architecture so that the learned model can transfer its weights to various frameworks and tasks.\r\n\r\nVATT linearly projects each modality into a feature vector and feeds it into a Transformer encoder. A semantically hierarchical common space is defined to account for the granularity of different modalities and noise contrastive estimation is employed to train the model.","description_state":"present","introduced_year":null,"introduced_by":{"title":"VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text","paper":"/paper/vatt-transformers-for-multimodal-self","first_author":"Hassan Akbari","n_authors":7,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/vatt-transformers-for-multimodal-self"},"source":{"url":"https://arxiv.org/abs/2104.11178v3","title":"VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Computer Vision","area_id":"computer-vision","collection":"Multi-Modal Methods","url":"/methods/category/multi-modal-methods","pwc_aliases":[]},{"area":"Computer Vision","area_id":"computer-vision","collection":"Vision Transformers","url":"/methods/category/vision-transformers","pwc_aliases":["vision-transformer"]}],"n_papers_tagged":3,"archive_num_papers":3,"papers_newest_first":[{"paper":"/paper/tell-what-you-hear-from-what-you-see-video-to","title":"Tell What You Hear From What You See -- Video to Audio Generation Through Text","date":"2024-11-08","arxiv_id":"2411.05679","n_code_links":1,"syntology":{"ran":0,"of":6,"unverified":6,"pointer_only":6}},{"paper":null,"title":"Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization","date":"2022-11-03","arxiv_id":"2211.02077","n_code_links":0,"syntology":null},{"paper":"/paper/vatt-transformers-for-multimodal-self","title":"VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text","date":"2021-04-22","arxiv_id":"2104.11178","n_code_links":5,"syntology":{"ran":5,"of":8,"unverified":3,"pointer_only":8}}],"papers_shown":3,"tasks":[{"task":"/task/action-classification","name":"Action Classification","papers":1},{"task":"/task/action-recognition-in-videos","name":"Action Recognition","papers":1},{"task":"/task/action-recognition-in-videos-2","name":"Action Recognition In Videos","papers":1},{"task":"/task/audio-classification","name":"Audio Classification","papers":1},{"task":"/task/audio-generation","name":"Audio Generation","papers":1},{"task":"/task/audio-captioning","name":"Audio captioning","papers":1},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":1},{"task":"/task/classification","name":"General Classification","papers":1},{"task":"/task/image-classification","name":"Image Classification","papers":1},{"task":"/task/retrieval","name":"Retrieval","papers":1},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":1},{"task":"/task/action-recognition","name":"Temporal Action Localization","papers":1},{"task":"/task/text-to-video-retrieval","name":"Text to Video Retrieval","papers":1},{"task":null,"name":"Triplet","papers":1},{"task":"/task/video-retrieval","name":"Video Retrieval","papers":1},{"task":"/task/video-to-sound-generation","name":"Video-to-Sound Generation","papers":1},{"task":"/task/zero-shot-video-retrieval","name":"Zero-Shot Video Retrieval","papers":1},{"task":"/task/image-classification","name":"image-classification","papers":1}],"tasks_shown":18,"n_tasks":18,"usage_by_year":[{"year":"2021","papers":1},{"year":"2022","papers":1},{"year":"2024","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/vatt"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}