{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-01967","title":"Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuning","arxiv_id":"2606.01967","date":"2026-06-01","proceeding":"ICML","authors":["Wenhang Shi","Yiren Chen","Shuqing Bian","Zhe Zhao","Jinhao Dong","Pengfei Hu","Wei Lu","Xiaoyong Du"],"abstract":"While prompt engineering is instrumental in maximizing the capabilities of Large Language Models (LLMs) during inference, the role of prompts during training remains critically underexplored. Prevailing fine-tuning paradigms typically treat training prompts as mere surface forms, assuming that semantically equivalent instructions yield identical learning outcomes. However, we reveal that this equivalence is deceptive: while paraphrased prompts often lead to comparable in-task performance, they induce drastically different cross-task impacts regarding catastrophic forgetting and generalization. Crucially, these impacts are positively correlated across tasks, indicating the existence of superior prompts that consistently yield better performance. Furthermore, we discover that these superior prompts can be robustly identified by task loss prior to learning. Leveraging these insights, we introduce State-Adaptive Prompt Optimization (SAPO), a lightweight yet effective training strategy that shifts task formulation from a static input to a dynamic, state-adaptive variable. Comprehensive experiments on diverse benchmarks confirm its effectiveness, which significantly mitigates forgetting while improving generalization, achieving substantial performance gains over state-of-the-art methods. These results provide insights into how training prompts shape learning dynamics and offer a practical recipe for robust fine-tuning. Our code is available at https://github.com/Eric8932/SAPO.","url_abs":"https://arxiv.org/abs/2606.01967","url_pdf":"https://arxiv.org/pdf/2606.01967","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.01967","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.01967"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Eric8932/SAPO","reach":null}],"summary":{"ran":3,"ran_draft_wrong":2,"ran_violates":2,"ran_fixture":1,"unverified":6},"by_repo_kind":{"found_in_text":{"samples":14,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"b0d4930635911b22","entry":"AdaptedAttention","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0d4930635911b22"}},{"code_sha256_prefix":"5dd8180f4190e417","entry":"AdaptionPromptConfig","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5dd8180f4190e417"}},{"code_sha256_prefix":"5e9f03c839ac9752","entry":"PeftConfigMixin","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e9f03c839ac9752"}},{"code_sha256_prefix":"127b2b488cacb9c4","entry":"_get_submodules","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"127b2b488cacb9c4"}},{"code_sha256_prefix":"56f9a78a6bf96dc2","entry":"is_adaption_prompt_trainable","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56f9a78a6bf96dc2"}},{"code_sha256_prefix":"1b3b0b10b6eb22b0","entry":"llama_apply_rotary_pos_emb","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b3b0b10b6eb22b0"}},{"code_sha256_prefix":"69890b767e2d4847","entry":"llama_compute_query_states","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69890b767e2d4847"}},{"code_sha256_prefix":"47805bd9ef4bfdd8","entry":"llama_rotate_half","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"47805bd9ef4bfdd8"}},{"code_sha256_prefix":"d717b0d33731ef3d","entry":"AdaptionPromptModel","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d717b0d33731ef3d"}},{"code_sha256_prefix":"39fd720bdfd54938","entry":"PeftConfig","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39fd720bdfd54938"}},{"code_sha256_prefix":"e18ccecc6524cf57","entry":"PeftType","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e18ccecc6524cf57"}},{"code_sha256_prefix":"fa1641c2e7bbbae1","entry":"TaskType","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa1641c2e7bbbae1"}},{"code_sha256_prefix":"599fe61280ffc433","entry":"_freeze_adapter","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"599fe61280ffc433"}},{"code_sha256_prefix":"b48f3b95aab0a0de","entry":"prepare_config","repo":"Eric8932/SAPO","repo_kind":"found_in_text","path":"src/peft_custom/tuners/adaption_prompt.py","file_url":"https://github.com/Eric8932/SAPO/blob/HEAD/src/peft_custom/tuners/adaption_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b48f3b95aab0a0de"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}