{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tuning-language-models-by-proxy","title":"Tuning Language Models by Proxy","arxiv_id":"2401.08565","date":"2024-01-16","proceeding":null,"authors":["Alisa Liu","Xiaochuang Han","Yizhong Wang","Yulia Tsvetkov","Yejin Choi","Noah A. Smith"],"abstract":"Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors. However, tuning these models has become increasingly resource-intensive, or impossible when model weights are private. We introduce proxy-tuning, a lightweight decoding-time algorithm that operates on top of black-box LMs to achieve the same end as direct tuning, but by accessing only its predictions over the output vocabulary, not its parameters. Our method tunes a smaller LM, then applies the difference between the predictions of the small tuned and untuned LMs to shift the original predictions of the larger untuned model in the direction of tuning, while retaining the benefits of larger-scale pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its truly-tuned chat version, when evaluated across knowledge, reasoning, and safety benchmarks. We then demonstrate the generality of proxy-tuning by applying it to domain adaptation on code, and task-specific finetuning on question-answering and math problems. Finally, we show how to proxy-tune a truly black-box LM, GPT-3.5, for temporal adaptation, increasing its knowledge about recent events. Our work demonstrates the promise of using small tuned LMs to efficiently customize large, potentially proprietary LMs through decoding-time guidance.","url_abs":"https://arxiv.org/abs/2401.08565v4","url_pdf":"https://arxiv.org/pdf/2401.08565v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tuning-language-models-by-proxy","repo_url":"https://github.com/alisawuffles/proxy-tuning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"tuning-language-models-by-proxy","repo_url":"https://github.com/ZHZisZZ/emulated-disalignment","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"math","task_name":"Math"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"truthfulqa","task_name":"TruthfulQA"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.08565","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.08565"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alisawuffles/proxy-tuning","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ZHZisZZ/emulated-disalignment","reach":null}],"summary":{"ran":6,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"8f75e8446b4e0440","entry":"add_pad_token","repo":"alisawuffles/proxy-tuning","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/alisawuffles/proxy-tuning/blob/HEAD/eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f75e8446b4e0440"}},{"code_sha256_prefix":"20ca9ae3db782b7e","entry":"create_prompt_with_llama2_chat_format","repo":"alisawuffles/proxy-tuning","repo_kind":"official","path":"eval/templates.py","file_url":"https://github.com/alisawuffles/proxy-tuning/blob/HEAD/eval/templates.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20ca9ae3db782b7e"}},{"code_sha256_prefix":"1cb0eeb16cf04ee3","entry":"encode_with_messages_format","repo":"alisawuffles/proxy-tuning","repo_kind":"official","path":"open_instruct/finetune.py","file_url":"https://github.com/alisawuffles/proxy-tuning/blob/HEAD/open_instruct/finetune.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cb0eeb16cf04ee3"}},{"code_sha256_prefix":"88b8bf1223d522a4","entry":"encode_with_prompt_completion_format","repo":"alisawuffles/proxy-tuning","repo_kind":"official","path":"open_instruct/finetune.py","file_url":"https://github.com/alisawuffles/proxy-tuning/blob/HEAD/open_instruct/finetune.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"88b8bf1223d522a4"}},{"code_sha256_prefix":"b304416cbba64cb7","entry":"generate_completions","repo":"alisawuffles/proxy-tuning","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/alisawuffles/proxy-tuning/blob/HEAD/eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b304416cbba64cb7"}},{"code_sha256_prefix":"a151b8d96cbc70e8","entry":"get_equation_lhs_rhs_indices","repo":"alisawuffles/proxy-tuning","repo_kind":"official","path":"analysis/gsm_analysis.py","file_url":"https://github.com/alisawuffles/proxy-tuning/blob/HEAD/analysis/gsm_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a151b8d96cbc70e8"}},{"code_sha256_prefix":"e0ffc0f49d6c9840","entry":"load_lm_and_tokenizer","repo":"alisawuffles/proxy-tuning","repo_kind":"official","path":"eval/utils.py","file_url":"https://github.com/alisawuffles/proxy-tuning/blob/HEAD/eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0ffc0f49d6c9840"}},{"code_sha256_prefix":"ec1ba7666d388b75","entry":"summarize_results","repo":"alisawuffles/proxy-tuning","repo_kind":"official","path":"analysis/utils.py","file_url":"https://github.com/alisawuffles/proxy-tuning/blob/HEAD/analysis/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec1ba7666d388b75"}},{"code_sha256_prefix":"11c267e7913fc8dd","entry":"flatten_batch_results","repo":"alisawuffles/proxy-tuning","repo_kind":"official","path":"analysis/utils.py","file_url":"https://github.com/alisawuffles/proxy-tuning/blob/HEAD/analysis/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"11c267e7913fc8dd"}},{"code_sha256_prefix":"776a712a353130e7","entry":"read_problems","repo":"alisawuffles/proxy-tuning","repo_kind":"official","path":"eval/codex_humaneval/data.py","file_url":"https://github.com/alisawuffles/proxy-tuning/blob/HEAD/eval/codex_humaneval/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"776a712a353130e7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}