Browse State-of-the-Art › Continual Pretraining
Continual Pretraining
34 papers with code · 3 benchmarks · 3 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
3 leaderboard tables shown for this task, 3 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
Syntology 1 newer paper was judged to report on this task's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): ACL-ARC (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| ACL-ARC (1 row) | DAS | Continual Pre-training of Language Models | code | — | Compare |
| AG News (1 row) | CPT | Continual Training of Language Models for Few-Shot Learning | code | — | Compare |
| SciERC (1 row) | DAS | Continual Pre-training of Language Models | code | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
3 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 70 · with a code link: 34 · where Syntology ran a sample: 12 (7 with a run with no instrument failure, 5 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (12 of 70 tagged: 7 with a run with no instrument failure, 5 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 70 of 70, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
11 Apr 2024 3 repositories listedUnlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution.
-
11 Oct 2022 3 repositories listedRecent work on applying large language models (LMs) achieves impressive performance in many NLP applications.
-
12 Feb 2024 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)We release our curated AutoMathText dataset to facilitate future research in automated domain-specific data curation.
-
27 Sep 2023 2 repositories listedWe also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts…
-
9 Feb 2023 2 repositories listed Syntology official (archive's flag): 1 ran · 2 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified (of 6 harvested samples)Geospatial technologies are becoming increasingly essential in our world for a wide range of applications, including agriculture, urban planning, and disaster response.
-
7 Feb 2023 2 repositories listedA novel proxy is also proposed to preserve the general knowledge in the original LM.
-
30 Dec 2020 2 repositories listedWhile pre-trained language models (PTLMs) have achieved noticeable success on many NLP tasks, they still struggle for tasks that require event temporal reasoning, which is essential for event-centric applications.
-
31 Jul 2020 2 repositories listedIn this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining…
-
30 May 2025 1 repository listedThe performance of large language models (LLMs) continues to improve, as reflected in rising scores on standard benchmarks.
-
22 May 2025 1 repository listedWe present a Japanese domain-specific language model for the pharmaceutical field, developed through continual pretraining on 2 billion Japanese pharmaceutical tokens and 8 billion English biomedical tokens.
-
2 Apr 2025 1 repository listedLarge Language Models (LLMs) trained on historical web data inevitably become outdated.
-
6 Mar 2025 1 repository listedOur watermarks are designed to be memorized by the LLM through seamlessly integrating in its training data, making them harder to detect lexically during preprocessing.
-
9 Jan 2025 1 repository listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 4 where Syntology's instrument failed) · 4 unverified (of 13 harvested samples) · 13 pointer-only (licence)Domain-adaptive post-training of large language models (LLMs) has emerged as a promising approach for specialized domains such as medicine and finance.
-
11 Dec 2024 1 repository listedThe integration of artificial intelligence (AI) in legal judgment prediction (LJP) has the potential to transform the legal landscape, particularly in jurisdictions like India, where a significant backlog of cases…
-
21 Oct 2024 1 repository listedExperimental results demonstrate the effectiveness of our approach, achieving a 5% absolute performance improvement on Leandojo benchmark.
-
26 Sep 2024 1 repository listedHowever, this removal increases the burden on token embeddings to encode all language-specific information, which may hinder the model's ability to produce more language-neutral representations.
-
26 Aug 2024 1 repository listedIn this work, we complement current perspectives on continual pretraining through a research test bed as well as provide comprehensive guidance for effective continual model updates in such scenarios.
-
18 Jul 2024 1 repository listedThis paper introduces long-context Granite code models that support effective context windows of up to 128K tokens.
-
10 Jun 2024 1 repository listedThis survey delves into the sophisticated landscape of lifelong learning, categorizing strategies into two primary groups: Internal Knowledge and External Knowledge.
-
30 May 2024 1 repository listedSecond, we revisit and explore cross-domain continual pretraining for both multispectral and SAR imagery, building efficient EO foundation models from strongest vision models such as DINOv2.
-
20 May 2024 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples) · 3 pointer-only (licence)Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models.
-
24 Apr 2024 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 1 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples) · 7 pointer-only (licence)Despite the recent progress in long-context language models, it remains elusive how transformer-based models exhibit the capability to retrieve relevant information from arbitrary locations within the long context.
-
7 Mar 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 6 unverified (of 8 harvested samples)The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models.
-
15 Feb 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)We demonstrate that continual pretraining of the full model on 1B-5B tokens of such data is an effective and affordable strategy for scaling the context length of language models to 128K.
-
11 Nov 2023 1 repository listed Syntology official (archive's flag): 1 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)The limited availability of labelled data in Action Quality Assessment (AQA), has forced previous works to fine-tune their models pretrained on large-scale domain-general datasets.
-
14 Aug 2023 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 1 violated, 7 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified (of 14 harvested samples) · 1 pointer-only (licence)Regarding the growing nature of real-world data, such an offline training paradigm on ever-expanding data is unsustainable, because models lack the continual learning ability to accumulate knowledge constantly.
-
1 Jan 2023 1 repository listedRegarding the growing nature of real-world data, such an offline training paradigm on ever-expanding data is unsustainable, because models lack the continual learning ability to accumulate knowledge constantly.
-
21 Nov 2022 1 repository listedContinual pretraining is a popular way of building a domain-specific pretrained language model from a general-domain language model.
-
8 Nov 2022 1 repository listedWe conducted experiments using our method on datasets with a large vocabulary gap from a source domain.
-
19 May 2022 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)We formalize and investigate the characteristics of the continual pre-training scenario in both language and vision environments, where a model is continually pre-trained on a stream of incoming data and only later…
-
22 Apr 2022 1 repository listed
-
6 Jan 2022 1 repository listed
-
16 Oct 2021 1 repository listed
-
10 Sep 2021 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified (of 9 harvested samples) · 9 pointer-only (licence)
-
Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content25 Jun 2025 0 repositories listed
-
LLaVA-c: Continual Improved Visual Instruction Tuning10 Jun 2025 0 repositories listed
-
Enhance Mobile Agents Thinking Process Via Iterative Preference Learning18 May 2025 0 repositories listed
-
Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning15 May 2025 0 repositories listed
-
Efficient Domain-adaptive Continual Pretraining for the Process Industry in the German Language28 Apr 2025 0 repositories listed
-
Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them27 Mar 2025 0 repositories listed
-
AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text24 Mar 2025 0 repositories listed
-
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling24 Mar 2025 0 repositories listed
-
Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models10 Dec 2024 0 repositories listed
-
DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining30 Sep 2024 0 repositories listed
-
The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Merging30 Sep 2024 0 repositories listed
-
AstroMLab 2: AstroLLaMA-2-70B Model and Benchmarking Specialised LLMs for Astronomy29 Sep 2024 0 repositories listed
-
Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach9 Sep 2024 0 repositories listed
-
RedWhale: An Adapted Korean LLM Through Efficient Continual Pretraining21 Aug 2024 0 repositories listed
-
Bilingual Adaptation of Monolingual Foundation Models13 Jul 2024 0 repositories listed
-
70B-parameter large language models in Japanese medical question-answering21 Jun 2024 0 repositories listed
-
Open Generative Large Language Models for Galician19 Jun 2024 0 repositories listed
-
Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective19 Jun 2024 0 repositories listed
-
BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLM17 Jun 2024 0 repositories listed
-
LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models2 Jun 2024 0 repositories listed
-
Cross-sensor self-supervised training and alignment for remote sensing16 May 2024 0 repositories listed
-
ChuXin: 1.6B Technical Report8 May 2024 0 repositories listed
-
Pretraining and Updates of Domain-Specific LLM: A Case Study in the Japanese Business Domain12 Apr 2024 0 repositories listed
-
CEM: A Data-Efficient Method for Large Language Models to Continue Evolving From Mistakes11 Apr 2024 0 repositories listed
-
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code30 Mar 2024 0 repositories listed
-
PARAMANU-AYN: Pretrain from scratch or Continual Pretraining of LLMs for Legal Domain Adaptation?20 Mar 2024 0 repositories listed
-
Investigating Continual Pretraining in Large Language Models: Insights and Implications27 Feb 2024 0 repositories listed
-
Continual Learning for Large Language Models: A Survey2 Feb 2024 0 repositories listed
-
RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization25 Jan 2024 0 repositories listed
-
DD-TIG at Constraint@ACL2022: Multimodal Understanding and Reasoning for Role Labeling of Entities in Hateful Memes1 May 2022 0 repositories listed
-
AdaPrompt: Adaptive Model Training for Prompt-based NLP10 Feb 2022 0 repositories listed
-
On the Robustness of Reading Comprehension Models to Entity Renaming16 Nov 2021 0 repositories listed
-
Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks1 Nov 2021 0 repositories listed
-
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora16 Oct 2021 0 repositories listed
-
Revisiting Pretraining with Adapters1 Aug 2021 0 repositories listed
Syntology lines on 12 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections