Browse State-of-the-Art › Unsupervised Pre-training
Unsupervised Pre-training
126 papers with code · 2 benchmarks · 7 datasets archive 2025-07-28
Pre-training a neural network using unsupervised (self-supervised) auxiliary tasks on unlabeled data.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| Measles (5 rows) | 15RDLs | — | — | — | Compare |
| UCI measles (3 rows) | (unnamed in the archive) | RMDL: Random Multimodel Deep Learning for Classification | code | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
7 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 265 · with a code link: 126 · where Syntology ran a sample: 43 (36 with a run with no instrument failure, 7 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (43 of 265 tagged: 36 with a run with no instrument failure, 7 where every run was a failure of Syntology's instrument)
Page 1 of 3: papers 1 to 100 of 265, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
11 Dec 2020 12 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 4 harvested samples) · 4 pointer-only (licence)We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning.
-
29 Jul 2019 7 repositories listed Syntology 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing.
-
11 Apr 2019 7 repositories listed Syntology community repositories only · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)Our experiments on WSJ reduce WER of a strong character-based log-mel filterbank baseline by up to 36% when only a few hours of transcribed data is available.
-
6 Oct 2020 6 repositories listed Syntology official (archive's flag): 16 ran · 23 ran (of which 0 constructed an object rather than computing a result; 19 with no instrument failure: 0 honoured, 0 violated, 19 with no contract checked; 4 where Syntology's instrument failed) · 9 unverified (of 32 harvested samples) · 6 pointer-only (licence)In this work we propose for the first time a transformer-based framework for unsupervised representation learning of multivariate time series.
-
30 Mar 2021 5 repositories listed Syntology official (archive's flag): 4 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 8 harvested samples)Transfer learning approaches can reduce the data requirements of deep learning algorithms.
-
9 Nov 2015 5 repositories listed Syntology community repositories only · 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 10 harvested samples)We propose ways to improve the performance of fully connected networks.
-
9 Aug 2020 4 repositories listedOur representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away.
-
31 Dec 2018 4 repositories listed Syntology official (archive's flag): 8 ran · 22 ran (of which 0 constructed an object rather than computing a result; 19 with no instrument failure: 0 honoured, 0 violated, 19 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified (of 28 harvested samples) · 3 pointer-only (licence)We show that constituency parsing benefits from unsupervised pre-training across a variety of languages and a range of pre-training conditions.
-
20 Jan 2024 3 repositories listed Syntology official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)In this study, we aim to shed light on this issue by revealing that masked auto-encoder (MAE) pre-training with enhanced decoding significantly improves the term coverage of input tokens in dense representations…
-
21 Dec 2020 3 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)With this in mind, we propose a teacher-student scheme to learn representations by training a convolutional net to reconstruct a bag-of-visual-words (BoW) representation of an image, given as input a perturbed version…
-
22 Oct 2020 3 repositories listedSelf-training and unsupervised pre-training have emerged as effective approaches to improve speech recognition systems using unlabeled data.
-
3 Aug 2020 3 repositories listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 9 harvested samples)In this paper, we compose a trilogy of exploring the basic and generic supervision in the sequence from spatial, spatiotemporal and sequential perspectives.
-
17 Oct 2024 2 repositories listed Syntology official: harvested, nothing ran · 10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 2 honoured, 2 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified (of 16 harvested samples) · 8 pointer-only (licence)Gaussian splatting and single/multi-view depth estimation are typically studied in isolation.
-
23 Feb 2024 2 repositories listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 4 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified (of 16 harvested samples) · 16 pointer-only (licence)While a number of methods have been proposed to enable generic self-supervised RL, based on principles such as goal-conditioned RL, behavioral cloning, and unsupervised skill learning, such methods remain limited in…
-
22 May 2023 2 repositories listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 3 pointer-only (licence)Semi-supervised learning (SSL) is a popular setting aiming to effectively utilize unlabelled data to improve model performance in downstream natural language processing (NLP) tasks.
-
2 May 2023 2 repositories listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 8 harvested samples) · 3 pointer-only (licence)Language models (LMs) trained on vast quantities of unlabelled data have greatly advanced the field of natural language processing (NLP).
-
21 Jul 2022 2 repositories listedWe find that our proposed pre-training methods help in modeling the data at a patient and population level and improve performance in different fine-tuning tasks on all datasets.
-
30 Mar 2022 2 repositories listed Syntology official (archive's flag): 2 ran · 5 ran (of which 2 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 4 pointer-only (licence)Since theses ID labels automatically derived from tracklets inevitably contain noises, we develop a large-scale Pre-training framework utilizing Noisy Labels (PNL), which consists of three learning modules: supervised…
-
25 Mar 2022 2 repositories listedOur framework consists of two phases: we pre-train an action-free latent video prediction model, and then utilize the pre-trained representations for efficiently learning action-conditional world models on unseen…
-
23 Mar 2022 2 repositories listedWe test our method on two medical datasets of patient records, TADPOLE and MIMIC-III, including imaging and non-imaging features and different prediction tasks.
-
25 Dec 2021 2 repositories listedIdentifying novel drug-target interactions (DTI) is a critical and rate limiting step in drug discovery.
-
17 Aug 2021 2 repositories listedIn particular, we propose to generate random layouts of a scene by making use of the objects in the synthetic CAD dataset and learn the 3D scene representation by applying object-level contrastive learning on two random…
-
15 Jun 2021 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples)In many control problems that include vision, optimal controls can be inferred from the location of the objects in the scene.
-
9 Jun 2021 2 repositories listed Syntology official (archive's flag): 4 ran · 7 ran (of which 1 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples) · 4 pointer-only (licence)We also show that our method is able to utilize real-time human feedback to effectively prevent reward exploitation and learn new behaviors that are difficult to specify with standard reward functions.
-
29 Apr 2021 2 repositories listedWe present a large-scale study on unsupervised spatiotemporal representation learning from videos.
-
30 Mar 2021 2 repositories listedRecent work on Deep Learning in the area of face analysis has focused on supervised learning for specific tasks of interest (e. g. face recognition, facial landmark localization etc.)
-
15 Feb 2021 2 repositories listedRecent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks.
-
11 Feb 2021 2 repositories listed Syntology official (archive's flag): 1 ran · 2 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 2 samples that ran constructed an object rather than computing a result (of 2 harvested samples) · 2 pointer-only (licence)To achieve this, we introduce a two-step framework that adopts a predetermined mid-level prior in a contrastive optimization objective to learn pixel embeddings.
-
2 Jan 2021 2 repositories listedWe also explore two approaches for end-to-end supervised training of the reader and retriever components in OpenQA models.
-
18 Nov 2020 2 repositories listedDEtection TRansformer (DETR) for object detection reaches competitive performance compared with Faster R-CNN via a transformer encoder-decoder architecture.
-
21 Jul 2020 2 repositories listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 2 pointer-only (licence)
-
4 May 2020 2 repositories listed
-
3 May 2019 2 repositories listed
-
7 Feb 2019 2 repositories listed
-
21 Nov 2015 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 3 pointer-only (licence)
-
20 Dec 2014 2 repositories listed
-
20 Dec 2013 2 repositories listed
-
5 Jun 2025 1 repository listed
-
2 Jun 2025 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 2 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples) · 2 pointer-only (licence)
-
10 Apr 2025 1 repository listed
-
3 Apr 2025 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 2 pointer-only (licence)
-
30 Mar 2025 1 repository listed
-
18 Mar 2025 1 repository listed
-
16 Dec 2024 1 repository listed
-
3 Oct 2024 1 repository listed
-
29 Sep 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
10 Aug 2024 1 repository listed
-
5 Aug 2024 1 repository listed
-
26 Jun 2024 1 repository listed
-
29 May 2024 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 1 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified (of 13 harvested samples) · 3 pointer-only (licence)
-
23 May 2024 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 3 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 1 pointer-only (licence)
-
22 May 2024 1 repository listed
-
21 May 2024 1 repository listed
-
17 May 2024 1 repository listed
-
4 May 2024 1 repository listed
-
29 Apr 2024 1 repository listed
-
8 Apr 2024 1 repository listed
-
26 Feb 2024 1 repository listed
-
6 Nov 2023 1 repository listed
-
13 Oct 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
5 Sep 2023 1 repository listed
-
22 Jul 2023 1 repository listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 4 honoured, 1 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 9 unverified (of 17 harvested samples) · 8 pointer-only (licence)
-
29 May 2023 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 9 harvested samples)
-
24 May 2023 1 repository listed
-
20 May 2023 1 repository listed
-
19 May 2023 1 repository listed
-
11 May 2023 1 repository listed
-
25 Apr 2023 1 repository listed
-
28 Mar 2023 1 repository listed
-
25 Mar 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples) · 1 pointer-only (licence)
-
11 Mar 2023 1 repository listed
-
11 Feb 2023 1 repository listed
-
19 Dec 2022 1 repository listed
-
26 Jul 2022 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
17 Jul 2022 1 repository listed Syntology 0 ran · 2 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Contextual embedding and model weighting by fusing domain knowledge on Biomedical Question Answering26 Jun 2022 1 repository listed
-
16 Jun 2022 1 repository listed
-
6 Jun 2022 1 repository listed
-
30 May 2022 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 2 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples)
-
19 Apr 2022 1 repository listed
-
11 Apr 2022 1 repository listed
-
25 Mar 2022 1 repository listed
-
14 Feb 2022 1 repository listed
-
17 Jan 2022 1 repository listed
-
17 Jan 2022 1 repository listed
-
9 Dec 2021 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
1 Dec 2021 1 repository listed
-
13 Nov 2021 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
13 Nov 2021 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)
-
1 Nov 2021 1 repository listed
-
14 Sep 2021 1 repository listed
-
6 Jun 2021 1 repository listed
-
31 May 2021 1 repository listed
-
3 May 2021 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples)
-
9 Apr 2021 1 repository listed
-
8 Mar 2021 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified (of 6 harvested samples)
-
5 Mar 2021 1 repository listed
-
14 Feb 2021 1 repository listed Syntology official (archive's flag): 9 ran · 9 ran (of which 7 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 10 harvested samples)
-
4 Feb 2021 1 repository listed
-
12 Jan 2021 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)
Syntology lines on 39 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections