Self-Supervised Learning

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

10 leaderboard tables shown for this task, 10 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

47 datasets whose archive record lists this task, ordered by the archive's paper count. 30 shown of 47 until expanded.

Subtasks archive 2025-07-28

2 subtasks in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 2,293 papers with code (5,044 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• A Simple Framework for Contrastive Learning of Visual Representations
  13 Feb 2020 96 repositories listed Syntology ran 79 of 137 samples · 58 unverified · 52 pointer-only (licence)
  This paper presents SimCLR: a simple framework for contrastive learning of visual representations.
• Masked Autoencoders Are Scalable Vision Learners
  11 Nov 2021 58 repositories listed Syntology ran 71 of 137 samples · 66 unverified · 73 pointer-only (licence)
  Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels.
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  26 Sep 2019 48 repositories listed Syntology ran 46 of 126 samples · 80 unverified · 22 pointer-only (licence)
  Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks.
• Emerging Properties in Self-Supervised Vision Transformers
  29 Apr 2021 32 repositories listed Syntology ran 5 of 20 samples · 15 unverified · 2 pointer-only (licence)
  In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets).
• Bootstrap your own latent: A new approach to self-supervised Learning
  13 Jun 2020 31 repositories listed Syntology ran 62 of 79 samples · 17 unverified · 46 pointer-only (licence)
  From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view.
• Supervised Contrastive Learning
  23 Apr 2020 26 repositories listed Syntology ran 6 of 23 samples · 17 unverified
  Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models.
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  20 Jun 2020 25 repositories listed Syntology ran 2 of 9 samples · 7 unverified · 2 pointer-only (licence)
  We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler.
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
  4 Mar 2021 24 repositories listed Syntology ran 21 of 26 samples · 5 unverified · 10 pointer-only (licence)
  This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors.
• TabNet: Attentive Interpretable Tabular Learning
  20 Aug 2019 19 repositories listed Syntology ran 1 of 17 samples · 16 unverified · 1 pointer-only (licence)
  We propose a novel high-performance and interpretable canonical deep tabular data learning architecture, TabNet.
• ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders
  2 Jan 2023 17 repositories listed Syntology ran 10 of 14 samples · 4 unverified · 9 pointer-only (licence)
  This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition…
• COVID-CT-Dataset: A CT Scan Dataset about COVID-19
  30 Mar 2020 17 repositories listed
  Using this dataset, we develop diagnosis methods based on multi-task learning and self-supervised learning, that achieve an F1 of 0.
• Digging Into Self-Supervised Monocular Depth Estimation
  4 Jun 2018 15 repositories listed Syntology ran 17 of 24 samples · 7 unverified · 6 pointer-only (licence)
  Per-pixel ground-truth depth data is challenging to acquire at scale.
• DeiT III: Revenge of the ViT
  14 Apr 2022 12 repositories listed
  Our evaluations on Image classification (ImageNet-1k with and without pre-training on ImageNet-21k), transfer learning and semantic segmentation show that our procedure outperforms by a large margin previous fully…
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
  7 Feb 2022 12 repositories listed Syntology ran 0 of 6 samples · 6 unverified
  While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind.
• Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images
  8 Jan 2021 12 repositories listed Syntology ran 12 of 13 samples · 1 unverified · 7 pointer-only (licence)
  In this paper, we present a very simple yet effective method named Neighbor2Neighbor to train an effective image denoising model with only noisy images.
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
  26 Oct 2021 9 repositories listed
  Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks.
• An Empirical Study of Training Self-Supervised Vision Transformers
  5 Apr 2021 9 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT.
• Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
  1 Dec 2020 8 repositories listed
  From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view.
• Charting the Right Manifold: Manifold Mixup for Few-shot Learning
  28 Jul 2019 8 repositories listed
  A recent regularization technique - Manifold Mixup focuses on learning a general-purpose representation, robust to small changes in the data distribution.
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
  19 Jan 2023 7 repositories listed Syntology ran 5 of 14 samples · 9 unverified · 13 pointer-only (licence)
  This paper demonstrates an approach for learning highly semantic image representations without relying on hand-crafted data-augmentations.
• BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learning
  1 Aug 2021 7 repositories listed Syntology ran 4 of 8 samples · 4 unverified · 2 pointer-only (licence)
  In particular, our BadEncoder injects backdoors into a pre-trained image encoder such that the downstream classifiers built based on the backdoored image encoder for different downstream tasks simultaneously inherit the…
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training
  18 Nov 2020 7 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  Compared to the baseline method MoCo-v2, our method introduces negligible computation overhead (only <1% slower), but demonstrates consistently superior performance when transferring to downstream dense prediction tasks…
• Whitening for Self-Supervised Representation Learning
  13 Jul 2020 7 repositories listed Syntology ran 10 of 13 samples · 3 unverified · 2 pointer-only (licence)
  Most of the current self-supervised representation learning (SSL) methods are based on the contrastive loss and the instance-discrimination task, where augmented versions of the same image instance ("positives") are…
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
  12 Jul 2020 7 repositories listed Syntology ran 5 of 14 samples · 9 unverified
  We present a large-scale comparison of various self-supervised models.
• Self-Supervised Learning of Pretext-Invariant Representations
  4 Dec 2019 7 repositories listed Syntology ran 3 of 4 samples · 1 unverified · 4 pointer-only (licence)
  The goal of self-supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations for a large training set of images.
• Time-Contrastive Networks: Self-Supervised Learning from Video
  23 Apr 2017 7 repositories listed
  While representations are learned from an unlabeled collection of task-related videos, robot behaviors such as pouring are learned by watching a single 3rd-person demonstration by a human.
• Context Autoencoder for Self-Supervised Representation Learning
  7 Feb 2022 6 repositories listed
  The pretraining tasks include two tasks: masked representation prediction - predict the representations for the masked patches, and masked patch reconstruction - reconstruct the masked patches.
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
  11 May 2021 6 repositories listed Syntology ran 11 of 17 samples · 6 unverified · 1 pointer-only (licence)
  Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image.
• Self-Supervised Learning with Swin Transformers
  10 May 2021 6 repositories listed
  We are witnessing a modeling shift from CNN to Transformers in computer vision.
• SUPERB: Speech processing Universal PERformance Benchmark
  3 May 2021 6 repositories listed Syntology ran 1 of 26 samples · 25 unverified
  SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data.

Syntology lines on 22 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.