Unsupervised Pre-training

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

7 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 126 papers with code (265 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• TabTransformer: Tabular Data Modeling Using Contextual Embeddings
  11 Dec 2020 12 repositories listed Syntology ran 1 of 4 samples · 3 unverified · 4 pointer-only (licence)
  We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning.
• Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
  29 Jul 2019 7 repositories listed Syntology ran 0 of 6 samples · 6 unverified
  Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing.
• wav2vec: Unsupervised Pre-training for Speech Recognition
  11 Apr 2019 7 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  Our experiments on WSJ reduce WER of a strong character-based log-mel filterbank baseline by up to 36% when only a few hours of transcribed data is available.
• A Transformer-based Framework for Multivariate Time Series Representation Learning
  6 Oct 2020 6 repositories listed Syntology ran 5 of 32 samples · 27 unverified · 2 pointer-only (licence)
  In this work we propose for the first time a transformer-based framework for unsupervised representation learning of multivariate time series.
• Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data
  30 Mar 2021 5 repositories listed Syntology ran 0 of 8 samples · 8 unverified
  Transfer learning approaches can reduce the data requirements of deep learning algorithms.
• How far can we go without convolution: Improving fully-connected networks
  9 Nov 2015 5 repositories listed Syntology ran 0 of 10 samples · 10 unverified
  We propose ways to improve the performance of fully connected networks.
• Spatiotemporal Contrastive Video Representation Learning
  9 Aug 2020 4 repositories listed
  Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away.
• Multilingual Constituency Parsing with Self-Attention and Pre-Training
  31 Dec 2018 4 repositories listed Syntology ran 3 of 28 samples · 25 unverified · 3 pointer-only (licence)
  We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions.
• Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval
  20 Jan 2024 3 repositories listed Syntology ran 2 of 2 samples · 0 unverified
  In this study, we aim to shed light on this issue by revealing that masked auto-encoder (MAE) pre-training with enhanced decoding significantly improves the term coverage of input tokens in dense representations,…
• OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learning
  21 Dec 2020 3 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  With this in mind, we propose a teacher-student scheme to learn representations by training a convolutional net to reconstruct a bag-of-visual-words (BoW) representation of an image, given as input a perturbed version…
• Self-training and Pre-training are Complementary for Speech Recognition
  22 Oct 2020 3 repositories listed
  Self-training and unsupervised pre-training have emerged as effective approaches to improve speech recognition systems using unlabeled data.
• SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning
  3 Aug 2020 3 repositories listed Syntology ran 2 of 9 samples · 7 unverified
  In this paper, we compose a trilogy of exploring the basic and generic supervision in the sequence from spatial, spatiotemporal and sequential perspectives.
• DepthSplat: Connecting Gaussian Splatting and Depth
  17 Oct 2024 2 repositories listed Syntology ran 8 of 16 samples · 8 unverified
  Gaussian splatting and single/multi-view depth estimation are typically studied in isolation.
• Foundation Policies with Hilbert Representations
  23 Feb 2024 2 repositories listed Syntology ran 0 of 16 samples · 16 unverified · 16 pointer-only (licence)
  While a number of methods have been proposed to enable generic self-supervised RL, based on principles such as goal-conditioned RL, behavioral cloning, and unsupervised skill learning, such methods remain limited in…
• Rethinking Semi-supervised Learning with Language Models
  22 May 2023 2 repositories listed Syntology ran 4 of 5 samples · 1 unverified
  Semi-supervised learning (SSL) is a popular setting aiming to effectively utilize unlabelled data to improve model performance in downstream natural language processing (NLP) tasks.
• Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner
  2 May 2023 2 repositories listed Syntology ran 8 of 8 samples · 0 unverified
  Language models (LMs) trained on vast quantities of unlabelled data have greatly advanced the field of natural language processing (NLP).
• Unsupervised pre-training of graph transformers on patient population graphs
  21 Jul 2022 2 repositories listed
  We find that our proposed pre-training methods help in modeling the data at a patient and population level and improve performance in different fine-tuning tasks on all datasets.
• Large-Scale Pre-training for Person Re-identification with Noisy Labels
  30 Mar 2022 2 repositories listed Syntology ran 5 of 5 samples · 0 unverified · 4 pointer-only (licence)
  Since theses ID labels automatically derived from tracklets inevitably contain noises, we develop a large-scale Pre-training framework utilizing Noisy Labels (PNL), which consists of three learning modules: supervised…
• Reinforcement Learning with Action-Free Pre-Training from Videos
  25 Mar 2022 2 repositories listed
  Our framework consists of two phases: we pre-train an action-free latent video prediction model, and then utilize the pre-trained representations for efficiently learning action-conditional world models on unseen…
• Unsupervised Pre-Training on Patient Population Graphs for Patient-Level Predictions
  23 Mar 2022 2 repositories listed
  We test our method on two medical datasets of patient records, TADPOLE and MIMIC-III, including imaging and non-imaging features and different prediction tasks.
• AI-Bind: Improving Binding Predictions for Novel Protein Targets and Ligands
  25 Dec 2021 2 repositories listed
  Identifying novel drug-target interactions (DTI) is a critical and rate limiting step in drug discovery.
• RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection
  17 Aug 2021 2 repositories listed
  In particular, we propose to generate random layouts of a scene by making use of the objects in the synthetic CAD dataset and learn the 3D scene representation by applying object-level contrastive learning on two random…
• Learning of feature points without additional supervision improves reinforcement learning from images
  15 Jun 2021 2 repositories listed Syntology ran 0 of 4 samples · 4 unverified
  In many control problems that include vision, optimal controls can be inferred from the location of the objects in the scene.
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
  9 Jun 2021 2 repositories listed Syntology ran 4 of 7 samples · 3 unverified · 3 pointer-only (licence)
  We also show that our method is able to utilize real-time human feedback to effectively prevent reward exploitation and learn new behaviors that are difficult to specify with standard reward functions.
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
  29 Apr 2021 2 repositories listed
  We present a large-scale study on unsupervised spatiotemporal representation learning from videos.
• Pre-training strategies and datasets for facial representation learning
  30 Mar 2021 2 repositories listed
  Recent work on Deep Learning in the area of face analysis has focused on supervised learning for specific tasks of interest (e.
• DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
  15 Feb 2021 2 repositories listed
  Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks.
• Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals
  11 Feb 2021 2 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  To achieve this, we introduce a two-step framework that adopts a predetermined mid-level prior in a contrastive optimization objective to learn pixel embeddings.
• End-to-End Training of Neural Retrievers for Open-Domain Question Answering
  2 Jan 2021 2 repositories listed
  We also explore two approaches for end-to-end supervised training of the reader and retriever components in OpenQA models.
• UP-DETR: Unsupervised Pre-training for Object Detection with Transformers
  18 Nov 2020 2 repositories listed
  DEtection TRansformer (DETR) for object detection reaches competitive performance compared with Faster R-CNN via a transformer encoder-decoder architecture.

Syntology lines on 18 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.