{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/eliciting-fine-tuned-transformer-capabilities","title":"Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques","arxiv_id":"2506.08060","date":"2025-06-09","proceeding":null,"authors":["Asankhaya Sharma"],"abstract":"Large language models have transformed natural language processing, yet supervised fine-tuning (SFT) remains computationally intensive. This paper formally proves that capabilities acquired through SFT can be approximated by a base transformer model using inference-time techniques, specifically in-context learning (ICL), without altering model parameters, under idealized assumptions including unbounded computational resources and access to the fine-tuning dataset. We extend these results to practical scenarios with finite context lengths and partial dataset access. For text generation tasks with fixed output length $l$, datasets of size $\\mathrm{O}\\left( \\frac{m V}{\\varepsilon^2} \\log \\frac{m}{\\delta} \\right)$ or, with bounded context, $\\mathrm{O}\\left( \\frac{l \\log V}{\\varepsilon^2} \\log \\frac{1}{\\delta} \\right)$ suffice to approximate fine-tuned behavior across $m$ contexts within error $\\varepsilon$, where $V$ is the vocabulary size and $\\delta$ is the failure probability. For linear classification, datasets of size $\\mathrm{O}\\left( \\frac{d}{\\varepsilon} \\right)$ or, with fixed context, $\\mathrm{O}\\left( \\frac{1}{\\varepsilon^2} \\log \\frac{1}{\\delta} \\right)$ are sufficient, where $d$ is the input dimension. Grounded in the Turing completeness of transformers, these results provide a theoretical foundation for resource-efficient deployment of large language models, with practical techniques like retrieval-augmented generation bridging theory to real-world applications.","url_abs":"https://arxiv.org/abs/2506.08060v1","url_pdf":"https://arxiv.org/pdf/2506.08060v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"eliciting-fine-tuned-transformer-capabilities","repo_url":"https://github.com/codelion/optillm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}