Papers › Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques
Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques
Asankhaya Sharma
Large language models have transformed natural language processing, yet supervised fine-tuning (SFT) remains computationally intensive. This paper formally proves that capabilities acquired through SFT can be approximated by a base transformer model using inference-time techniques, specifically in-context learning (ICL), without altering model parameters, under idealized assumptions including unbounded computational resources and access to the fine-tuning dataset. We extend these results to practical scenarios with finite context lengths and partial dataset access. For text generation tasks with fixed output length l, datasets of size O( (m V)/ε² logm/δ ) or, with bounded context, O( (l logV)/ε² log1/δ ) suffice to approximate fine-tuned behavior across m contexts within error ε, where V is the vocabulary size and δ is the failure probability. For linear classification, datasets of size O( d/ε ) or, with fixed context, O( 1/ε² log1/δ ) are sufficient, where d is the input dimension. Grounded in the Turing completeness of transformers, these results provide a theoretical foundation for resource-efficient deployment of large language models, with practical techniques like retrieval-augmented generation bridging theory to real-world applications.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections