Methods › Computer Vision › Multi-Modal Methods › VATT
VATT
Introduced by Hassan Akbari et al. in VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Video-Audio-Text Transformer, or VATT, is a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, it takes raw signals as inputs and extracts multidimensional representations that are rich enough to benefit a variety of downstream tasks. VATT borrows the exact architecture from BERT and ViT except the layer of tokenization and linear projection reserved for each modality separately. The design follows the same spirit as ViT that makes the minimal changes to the architecture so that the learned model can transfer its weights to various frameworks and tasks.
VATT linearly projects each modality into a feature vector and feeds it into a Transformer encoder. A semantically hierarchical common space is defined to account for the granularity of different modalities and noise contrastive estimation is employed to train the model.
Papers archive 2025-07-28
3 shown of 3, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Tell What You Hear From What You See -- Video to Audio Generation Through Text 8 Nov 2024 · 1 repository · arXiv:2411.05679Syntology ran 0 of 6 samples · 6 unverified · 6 pointer-only (licence)
-
Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization 3 Nov 2022 · 0 repositories · arXiv:2211.02077
-
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text 22 Apr 2021 · 5 repositories · arXiv:2104.11178Syntology ran 5 of 8 samples · 3 unverified · 8 pointer-only (licence)
Tasks archive 2025-07-28
18 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections