{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/spt-code-sequence-to-sequence-pre-training","title":"SPT-Code: Sequence-to-Sequence Pre-Training for Learning Source Code Representations","arxiv_id":"2201.01549","date":"2022-01-05","proceeding":null,"authors":["Changan Niu","Chuanyi Li","Vincent Ng","Jidong Ge","Liguo Huang","Bin Luo"],"abstract":"Recent years have seen the successful application of large pre-trained models to code representation learning, resulting in substantial improvements on many code-related downstream tasks. But there are issues surrounding their application to SE tasks. First, the majority of the pre-trained models focus on pre-training only the encoder of the Transformer. For generation tasks that are addressed using models with the encoder-decoder architecture, however, there is no reason why the decoder should be left out during pre-training. Second, many existing pre-trained models, including state-of-the-art models such as T5-learning, simply reuse the pre-training tasks designed for natural languages. Moreover, to learn the natural language description of source code needed eventually for code-related tasks such as code summarization, existing pre-training tasks require a bilingual corpus composed of source code and the associated natural language description, which severely limits the amount of data for pre-training. To this end, we propose SPT-Code, a sequence-to-sequence pre-trained model for source code. In order to pre-train SPT-Code in a sequence-to-sequence manner and address the aforementioned weaknesses associated with existing pre-training tasks, we introduce three pre-training tasks that are specifically designed to enable SPT-Code to learn knowledge of source code, the corresponding code structure, as well as a natural language description of the code without relying on any bilingual corpus, and eventually exploit these three sources of information when it is applied to downstream tasks. Experimental results demonstrate that SPT-Code achieves state-of-the-art performance on five code-related downstream tasks after fine-tuning.","url_abs":"https://arxiv.org/abs/2201.01549v4","url_pdf":"https://arxiv.org/pdf/2201.01549v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"spt-code-sequence-to-sequence-pre-training","repo_url":"https://github.com/nougatca/spt-code","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2201.01549","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2201.01549"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nougatca/spt-code","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6ff490d86a628905","entry":"avg_ir_metrics","repo":"nougatca/spt-code","repo_kind":"official","path":"sources/eval/metrics.py","file_url":"https://github.com/nougatca/spt-code/blob/HEAD/sources/eval/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ff490d86a628905"}},{"code_sha256_prefix":"37d5e92e475ada09","entry":"inputs_to_cuda","repo":"nougatca/spt-code","repo_kind":"official","path":"sources/models/utils.py","file_url":"https://github.com/nougatca/spt-code/blob/HEAD/sources/models/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37d5e92e475ada09"}},{"code_sha256_prefix":"9ccdbe15b1958f3a","entry":"ir_metrics","repo":"nougatca/spt-code","repo_kind":"official","path":"sources/eval/metrics.py","file_url":"https://github.com/nougatca/spt-code/blob/HEAD/sources/eval/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ccdbe15b1958f3a"}},{"code_sha256_prefix":"14a264bdefb0717c","entry":"remove_white_characters","repo":"nougatca/spt-code","repo_kind":"official","path":"sources/eval/metrics.py","file_url":"https://github.com/nougatca/spt-code/blob/HEAD/sources/eval/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14a264bdefb0717c"}},{"code_sha256_prefix":"1646803daed79ebe","entry":"run_eval","repo":"nougatca/spt-code","repo_kind":"official","path":"sources/downstream_tasks/search.py","file_url":"https://github.com/nougatca/spt-code/blob/HEAD/sources/downstream_tasks/search.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1646803daed79ebe"}},{"code_sha256_prefix":"2ac804bb1bee5821","entry":"transfer_arg_name","repo":"nougatca/spt-code","repo_kind":"official","path":"sources/args.py","file_url":"https://github.com/nougatca/spt-code/blob/HEAD/sources/args.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2ac804bb1bee5821"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}