{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/all-nlp-tasks-are-generation-tasks-a-general","title":"GLM: General Language Model Pretraining with Autoregressive Blank Infilling","arxiv_id":"2103.10360","date":"2021-03-18","proceeding":"ACL 2022 5","authors":["Zhengxiao Du","Yujie Qian","Xiao Liu","Ming Ding","Jiezhong Qiu","Zhilin Yang","Jie Tang"],"abstract":"There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). However, none of the pretraining frameworks performs the best for all tasks of three main categories including natural language understanding (NLU), unconditional generation, and conditional generation. We propose a General Language Model (GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25x parameters of BERT Large , demonstrating its generalizability to different downstream tasks.","url_abs":"https://arxiv.org/abs/2103.10360v2","url_pdf":"https://arxiv.org/pdf/2103.10360v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"all-nlp-tasks-are-generation-tasks-a-general","repo_url":"https://github.com/THUDM/GLM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"all-nlp-tasks-are-generation-tasks-a-general","repo_url":"https://github.com/BBuf/GLM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"all-nlp-tasks-are-generation-tasks-a-general","repo_url":"https://github.com/thudm/chatglm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"all-nlp-tasks-are-generation-tasks-a-general","repo_url":"https://github.com/thudm/chatglm2-6b","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"all-nlp-tasks-are-generation-tasks-a-general","repo_url":"https://github.com/thudm/chatglm3","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"all-nlp-tasks-are-generation-tasks-a-general","repo_url":"https://github.com/thudm/swissarmytransformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"all-nlp-tasks-are-generation-tasks-a-general","repo_url":"https://github.com/thudm/visualglm-6b","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"all-nlp-tasks-are-generation-tasks-a-general","repo_url":"https://github.com/jackaduma/ChatGLM-LoRA-RLHF-PyTorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"abstractive-text-summarization","task_name":"Abstractive Text Summarization"},{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"document-summarization","task_name":"Document Summarization"},{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/abstractive-text-summarization-on-cnn-daily","task":"Abstractive Text Summarization","dataset":"CNN / Daily Mail","model":"GLM-XXLarge","rank_in_archive_order":9,"of":53,"metrics":{"ROUGE-1":"44.7","ROUGE-2":"21.4","ROUGE-L":"41.4"},"uses_additional_data":true},{"leaderboard":"/sota/document-summarization-on-cnn-daily-mail","task":"Document Summarization","dataset":"CNN / Daily Mail","model":"GLM-XXLarge","rank_in_archive_order":4,"of":26,"metrics":{"ROUGE-1":"44.7","ROUGE-2":"21.4","ROUGE-L":"41.4"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GLM-XXLarge (bidirectional)","rank_in_archive_order":21,"of":37,"metrics":{"Accuracy":"72.35"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"GLM-XXLarge (unidirectional)","rank_in_archive_order":27,"of":37,"metrics":{"Accuracy":"67.18"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"GLM-XXLarge (bidirectional)","rank_in_archive_order":4,"of":89,"metrics":{"Number of params":"10000M","Test perplexity":"11.33"},"uses_additional_data":true},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"GLM-XXLarge (unidirectional)","rank_in_archive_order":5,"of":89,"metrics":{"Number of params":"10000M","Test perplexity":"12.22"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.10360","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.10360"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/chatglm2-6b","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jackaduma/ChatGLM-LoRA-RLHF-PyTorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/chatglm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/visualglm-6b","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/THUDM/GLM","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/swissarmytransformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BBuf/GLM","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/chatglm3","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"562eb223eab20399","entry":"get_masks_and_position_ids_glm","repo":"thudm/swissarmytransformer","repo_kind":"listed","path":"examples/glm/inference_glm.py","file_url":"https://github.com/thudm/swissarmytransformer/blob/HEAD/examples/glm/inference_glm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"562eb223eab20399"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}