{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coco-lm-correcting-and-contrasting-text","title":"COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining","arxiv_id":"2102.08473","date":"2021-02-16","proceeding":"NeurIPS 2021 12","authors":["Yu Meng","Chenyan Xiong","Payal Bajaj","Saurabh Tiwary","Paul Bennett","Jiawei Han","Xia Song"],"abstract":"We present a self-supervised learning framework, COCO-LM, that pretrains Language Models by COrrecting and COntrasting corrupted text sequences. Following ELECTRA-style pretraining, COCO-LM employs an auxiliary language model to corrupt text sequences, upon which it constructs two new tasks for pretraining the main model. The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics. The second sequence-level task, Sequence Contrastive Learning, is to align text sequences originated from the same source input while ensuring uniformity in the representation space. Experiments on GLUE and SQuAD demonstrate that COCO-LM not only outperforms recent state-of-the-art pretrained models in accuracy, but also improves pretraining efficiency. It achieves the MNLI accuracy of ELECTRA with 50% of its pretraining GPU hours. With the same pretraining steps of standard base/large-sized models, COCO-LM outperforms the previous best models by 1+ GLUE average points.","url_abs":"https://arxiv.org/abs/2102.08473v2","url_pdf":"https://arxiv.org/pdf/2102.08473v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coco-lm-correcting-and-contrasting-text","repo_url":"https://github.com/microsoft/coco-lm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"coco-lm-correcting-and-contrasting-text","repo_url":"https://github.com/lucidrains/coco-lm-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"electra","method_name":"ELECTRA"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2102.08473","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.08473"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/coco-lm-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/coco-lm","reach":{"status":"unanswered"}}],"summary":{"ran_draft_wrong":4,"ran":1,"unverified":1},"by_repo_kind":{"listed":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0897c18b5724ae22","entry":"gumbel_noise","repo":"lucidrains/coco-lm-pytorch","repo_kind":"listed","path":"coco_lm_pytorch/coco_lm_pytorch.py","file_url":"https://github.com/lucidrains/coco-lm-pytorch/blob/HEAD/coco_lm_pytorch/coco_lm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0897c18b5724ae22"}},{"code_sha256_prefix":"e104b05e0dac69a6","entry":"HiddenLayerExtractor","repo":"lucidrains/coco-lm-pytorch","repo_kind":"listed","path":"coco_lm_pytorch/coco_lm_pytorch.py","file_url":"https://github.com/lucidrains/coco-lm-pytorch/blob/HEAD/coco_lm_pytorch/coco_lm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e104b05e0dac69a6"}},{"code_sha256_prefix":"5926a238cd968779","entry":"log","repo":"lucidrains/coco-lm-pytorch","repo_kind":"listed","path":"coco_lm_pytorch/coco_lm_pytorch.py","file_url":"https://github.com/lucidrains/coco-lm-pytorch/blob/HEAD/coco_lm_pytorch/coco_lm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5926a238cd968779"}},{"code_sha256_prefix":"8774449259dd9b86","entry":"norm","repo":"lucidrains/coco-lm-pytorch","repo_kind":"listed","path":"coco_lm_pytorch/coco_lm_pytorch.py","file_url":"https://github.com/lucidrains/coco-lm-pytorch/blob/HEAD/coco_lm_pytorch/coco_lm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8774449259dd9b86"}},{"code_sha256_prefix":"81e946165f36f51e","entry":"norm","repo":"lucidrains/coco-lm-pytorch","repo_kind":"listed","path":"coco_lm_pytorch/coco_lm_pytorch.py","file_url":"https://github.com/lucidrains/coco-lm-pytorch/blob/HEAD/coco_lm_pytorch/coco_lm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81e946165f36f51e"}},{"code_sha256_prefix":"39c33ce01733622c","entry":"COCO","repo":"lucidrains/coco-lm-pytorch","repo_kind":"listed","path":"coco_lm_pytorch/coco_lm_pytorch.py","file_url":"https://github.com/lucidrains/coco-lm-pytorch/blob/HEAD/coco_lm_pytorch/coco_lm_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39c33ce01733622c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}