Papers › PALM: Pre-training an Autoencoding&Autoregressive Language Model for...

PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation

14 Apr 2020arXiv:2004.07159archive 2025-07-28

Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, Luo Si

Self-supervised pre-training, such as BERT, MASS and BART, has emerged as a powerful technique for natural language understanding and generation. Existing pre-training techniques employ autoencoding and/or autoregressive objectives to train Transformer-based models by recovering original word tokens from corrupted text with some masked tokens. The training goals of existing techniques are often inconsistent with the goals of many language generation tasks, such as generative question answering and conversational response generation, for producing new text given context. This work presents PALM with a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus, specifically designed for generating new text conditioned on context. The new scheme alleviates the mismatch introduced by the existing denoising scheme between pre-training and fine-tuning where generation is more than reconstructing original text. An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

alibaba/AliceMind officialpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to StructuralLM: Structural Pre-training for Form Understanding (arXiv:2105.11210; that paper's own run record: 0 ran · 5 unverified) and Parameter-Efficient Sparsity for Large Language Models Fine-Tuning (arXiv:2205.11005; that paper's own run record: 2 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 2 samples that ran constructed an object rather than computing a result); nothing checks that they implement this paper's method. report
overwindows/PALM pytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Abstractive Text SummarizationConversational Response GenerationDenoisingGenerative Question AnsweringLanguage ModelingLanguage ModellingNatural Language UnderstandingQuestion AnsweringQuestion GenerationQuestion-GenerationResponse GenerationText GenerationText Summarization

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Abstractive Text Summarization CNN / Daily Mail PALM ROUGE-1 44.30 #16 of 53 Archive leaderboard report
Abstractive Text Summarization CNN / Daily Mail PALM ROUGE-2 21.12 #16 of 53 Archive leaderboard report
Abstractive Text Summarization CNN / Daily Mail PALM ROUGE-L 41.41 #16 of 53 Archive leaderboard report
Text Generation CNN/Daily Mail PALM ROUGE-L 41.41 #1 of 1 Archive leaderboard report
Text Summarization GigaWord PALM ROUGE-1 39.45 #11 of 41 Archive leaderboard report
Text Summarization GigaWord PALM ROUGE-2 20.37 #11 of 41 Archive leaderboard report
Text Summarization GigaWord PALM ROUGE-L 36.75 #11 of 41 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

AdamAttentionAttention DropoutBARTBERTBPEDense ConnectionsDropoutLayer NormalizationLinear LayerLinear Warmup With Linear DecayMulti-Head AttentionResidual ConnectionSoftmaxWeight DecayWordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections