{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data2vec-a-general-framework-for-self-1","title":"data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language","arxiv_id":"2202.03555","date":"2022-02-07","proceeding":"Preprint 2022 1","authors":["Alexei Baevski","Wei-Ning Hsu","Qiantong Xu","Arun Babu","Jiatao Gu","Michael Auli"],"abstract":"While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind. To get us closer to general self-supervised learning, we present data2vec, a framework that uses the same learning method for either speech, NLP or computer vision. The core idea is to predict latent representations of the full input data based on a masked view of the input in a self-distillation setup using a standard Transformer architecture. Instead of predicting modality-specific targets such as words, visual tokens or units of human speech which are local in nature, data2vec predicts contextualized latent representations that contain information from the entire input. Experiments on the major benchmarks of speech recognition, image classification, and natural language understanding demonstrate a new state of the art or competitive performance to predominant approaches.","url_abs":"https://arxiv.org/abs/2202.03555v3","url_pdf":"https://arxiv.org/pdf/2202.03555v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/pytorch/fairseq/tree/main/examples/data2vec","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/AryanShekarlaban/data2vec-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/Guillem96/data2vec-vision","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/aau-es-ml/ssl_noise-robust_kws","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/ashutosh1919/data2vec-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/gatech-eic/s3-router","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/holgerbovbjerg/data2vec-kws","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/2024-MindSpore-1/Code2/tree/main/model-1/data2vec","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/MindCode-4/code-3/tree/main/data2vec","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/SPEECHCOG/data2vec_maiju","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://github.com/huggingface/transformers/tree/main/src/transformers/models/data2vec","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null},{"paper_slug":"data2vec-a-general-framework-for-self-1","repo_url":"https://gitlab.com/birder/birder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"linguistic-acceptability","task_name":"Linguistic Acceptability"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"paraphrase-identification","task_name":"Paraphrase Identification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"data2vec (ViT-H)","rank_in_archive_order":135,"of":1060,"metrics":{"Number of params":"656M","Top 1 Accuracy":"86.6%"},"uses_additional_data":false},{"leaderboard":"/sota/linguistic-acceptability-on-cola","task":"Linguistic Acceptability","dataset":"CoLA","model":"data2vec","rank_in_archive_order":30,"of":43,"metrics":{"Accuracy":"60.3%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-qnli","task":"Natural Language Inference","dataset":"QNLI","model":"data2vec","rank_in_archive_order":30,"of":43,"metrics":{"Accuracy":"91.1%"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-inference-on-rte","task":"Natural Language Inference","dataset":"RTE","model":"data2vec","rank_in_archive_order":55,"of":90,"metrics":{"Accuracy":"69.9%"},"uses_additional_data":false},{"leaderboard":"/sota/paraphrase-identification-on-quora-question","task":"Paraphrase Identification","dataset":"Quora Question Pairs","model":"data2vec","rank_in_archive_order":16,"of":31,"metrics":{"Accuracy":"92.4"},"uses_additional_data":false},{"leaderboard":"/sota/speech-recognition-on-librispeech-test-other","task":"Speech Recognition","dataset":"LibriSpeech test-other","model":"data2vec","rank_in_archive_order":12,"of":53,"metrics":{"Word Error Rate (WER)":"3.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.03555","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.03555"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aau-es-ml/ssl_noise-robust_kws","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Guillem96/data2vec-vision","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huggingface/transformers/tree/main/src/transformers/models/data2vec","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pytorch/fairseq/tree/main/examples/data2vec","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gatech-eic/s3-router","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/2024-MindSpore-1/Code2/tree/main/model-1/data2vec","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/holgerbovbjerg/data2vec-kws","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SPEECHCOG/data2vec_maiju","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://gitlab.com/birder/birder","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ashutosh1919/data2vec-pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-3/tree/main/data2vec","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AryanShekarlaban/data2vec-pytorch","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"listed":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2aa6bb5bf9a3cec5","entry":"compute_var","repo":"holgerbovbjerg/data2vec-kws","repo_kind":"listed","path":"data2vec/data2vec_utils/trainer.py","file_url":"https://github.com/holgerbovbjerg/data2vec-kws/blob/HEAD/data2vec/data2vec_utils/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2aa6bb5bf9a3cec5"}},{"code_sha256_prefix":"098fbbfa32b9e2f9","entry":"generate_masked_tensor","repo":"holgerbovbjerg/data2vec-kws","repo_kind":"listed","path":"data2vec/masking.py","file_url":"https://github.com/holgerbovbjerg/data2vec-kws/blob/HEAD/data2vec/masking.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"098fbbfa32b9e2f9"}},{"code_sha256_prefix":"d203ddadc113d9d6","entry":"get_config","repo":"holgerbovbjerg/data2vec-kws","repo_kind":"listed","path":"config_parser.py","file_url":"https://github.com/holgerbovbjerg/data2vec-kws/blob/HEAD/config_parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d203ddadc113d9d6"}},{"code_sha256_prefix":"6a3668fb4b8c0404","entry":"get_train_val_test_split","repo":"holgerbovbjerg/data2vec-kws","repo_kind":"listed","path":"utils/dataset.py","file_url":"https://github.com/holgerbovbjerg/data2vec-kws/blob/HEAD/utils/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a3668fb4b8c0404"}},{"code_sha256_prefix":"e8760411566b7fe1","entry":"kwt_from_name","repo":"holgerbovbjerg/data2vec-kws","repo_kind":"listed","path":"models/KWT.py","file_url":"https://github.com/holgerbovbjerg/data2vec-kws/blob/HEAD/models/KWT.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e8760411566b7fe1"}},{"code_sha256_prefix":"4ad2bd1c77f9444c","entry":"variance_loss","repo":"holgerbovbjerg/data2vec-kws","repo_kind":"listed","path":"data2vec/data2vec_utils/trainer.py","file_url":"https://github.com/holgerbovbjerg/data2vec-kws/blob/HEAD/data2vec/data2vec_utils/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ad2bd1c77f9444c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}