{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-power-of-scale-for-parameter-efficient","title":"The Power of Scale for Parameter-Efficient Prompt Tuning","arxiv_id":"2104.08691","date":"2021-04-18","proceeding":"EMNLP 2021 11","authors":["Brian Lester","Rami Al-Rfou","Noah Constant"],"abstract":"In this work, we explore \"prompt tuning\", a simple yet effective mechanism for learning \"soft prompts\" to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signal from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3's \"few-shot\" learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method \"closes the gap\" and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant in that large models are costly to share and serve, and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed \"prefix tuning\" of Li and Liang (2021), and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning.","url_abs":"https://arxiv.org/abs/2104.08691v2","url_pdf":"https://arxiv.org/pdf/2104.08691v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/google-research/prompt-tuning","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/RossNordby/SoftPromptsForEvaluation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/arazd/ProgressivePrompts","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/corolla-johnson/mkultra","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/exelents/soft-prompt-tuning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/google-deepmind/thunnini","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/hellokevin07/elastictrainer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/kipgparker/soft-prompt-tuning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/mkshing/Prompt-Tuning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/zhanglab-aim/lingo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"the-power-of-scale-for-parameter-efficient","repo_url":"https://github.com/Zeng-WH/Prompt-Tuning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.08691","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08691"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/thunnini","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/exelents/soft-prompt-tuning","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mkshing/Prompt-Tuning","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/prompt-tuning","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RossNordby/SoftPromptsForEvaluation","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/corolla-johnson/mkultra","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhanglab-aim/lingo","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Zeng-WH/Prompt-Tuning","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kipgparker/soft-prompt-tuning","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hellokevin07/elastictrainer","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/arazd/ProgressivePrompts","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":15},"by_repo_kind":{"listed":{"samples":15,"ran":0,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16ba263208ecf384","entry":"core_dummy_initialized","repo":"VE-FORBRYDERNE/mtj-softtuner","repo_kind":"listed","path":"mtj_softtuner/testing_utils.py","file_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner/blob/HEAD/mtj_softtuner/testing_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"16ba263208ecf384"}},{"code_sha256_prefix":"3be93347d7ab5153","entry":"core_partly_initialized","repo":"VE-FORBRYDERNE/mtj-softtuner","repo_kind":"listed","path":"mtj_softtuner/testing_utils.py","file_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner/blob/HEAD/mtj_softtuner/testing_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3be93347d7ab5153"}},{"code_sha256_prefix":"659a594a04601c37","entry":"core_test_mode","repo":"VE-FORBRYDERNE/mtj-softtuner","repo_kind":"listed","path":"mtj_softtuner/testing_utils.py","file_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner/blob/HEAD/mtj_softtuner/testing_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"659a594a04601c37"}},{"code_sha256_prefix":"51944f53542b7e19","entry":"get_config_from_root","repo":"VE-FORBRYDERNE/mtj-softtuner","repo_kind":"listed","path":"versioneer.py","file_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner/blob/HEAD/versioneer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51944f53542b7e19"}},{"code_sha256_prefix":"04adf1e29b3ea4b7","entry":"get_model_config_tokenizer","repo":"exelents/soft-prompt-tuning","repo_kind":"listed","path":"gptj_utils.py","file_url":"https://github.com/exelents/soft-prompt-tuning/blob/HEAD/gptj_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04adf1e29b3ea4b7"}},{"code_sha256_prefix":"641dddbb7b534717","entry":"is_time_to_exit","repo":"exelents/soft-prompt-tuning","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/exelents/soft-prompt-tuning/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"641dddbb7b534717"}},{"code_sha256_prefix":"e386d3a24af4d168","entry":"label_smoothed_nll_loss","repo":"exelents/soft-prompt-tuning","repo_kind":"listed","path":"utils1.py","file_url":"https://github.com/exelents/soft-prompt-tuning/blob/HEAD/utils1.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e386d3a24af4d168"}},{"code_sha256_prefix":"7c43af131a6f39a6","entry":"lmap","repo":"exelents/soft-prompt-tuning","repo_kind":"listed","path":"utils1.py","file_url":"https://github.com/exelents/soft-prompt-tuning/blob/HEAD/utils1.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c43af131a6f39a6"}},{"code_sha256_prefix":"29fe77a79233bc67","entry":"normalize_chess_elos","repo":"RossNordby/SoftPromptsForEvaluation","repo_kind":"listed","path":"chess_training/dataset_loading.py","file_url":"https://github.com/RossNordby/SoftPromptsForEvaluation/blob/HEAD/chess_training/dataset_loading.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29fe77a79233bc67"}},{"code_sha256_prefix":"c55cafde08716f01","entry":"register_vcs_handler","repo":"VE-FORBRYDERNE/mtj-softtuner","repo_kind":"listed","path":"mtj_softtuner/_version.py","file_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner/blob/HEAD/mtj_softtuner/_version.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c55cafde08716f01"}},{"code_sha256_prefix":"9dc618d67058e3e6","entry":"register_vcs_handler","repo":"VE-FORBRYDERNE/mtj-softtuner","repo_kind":"listed","path":"versioneer.py","file_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner/blob/HEAD/versioneer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9dc618d67058e3e6"}},{"code_sha256_prefix":"e4eaafa9797969b1","entry":"restore_variable","repo":"VE-FORBRYDERNE/mtj-softtuner","repo_kind":"listed","path":"mtj_softtuner/serialization.py","file_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner/blob/HEAD/mtj_softtuner/serialization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4eaafa9797969b1"}},{"code_sha256_prefix":"dbb37355631dabcf","entry":"run_command","repo":"VE-FORBRYDERNE/mtj-softtuner","repo_kind":"listed","path":"mtj_softtuner/_version.py","file_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner/blob/HEAD/mtj_softtuner/_version.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dbb37355631dabcf"}},{"code_sha256_prefix":"33d302ec4e7efc18","entry":"shatter","repo":"VE-FORBRYDERNE/mtj-softtuner","repo_kind":"listed","path":"mtj_softtuner/core.py","file_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner/blob/HEAD/mtj_softtuner/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"33d302ec4e7efc18"}},{"code_sha256_prefix":"5032e3b81c0fffb3","entry":"versions_from_parentdir","repo":"VE-FORBRYDERNE/mtj-softtuner","repo_kind":"listed","path":"mtj_softtuner/_version.py","file_url":"https://github.com/VE-FORBRYDERNE/mtj-softtuner/blob/HEAD/mtj_softtuner/_version.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5032e3b81c0fffb3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}