{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reasoning-on-a-budget-miniaturizing-deepseek","title":"Reasoning on a Budget: Miniaturizing DeepSeek R1 with SFT-GRPO Alignment for Instruction-Tuned LLMs","arxiv_id":null,"date":"2025-05-16","proceeding":"techrxiv 2025 5","authors":["Esmaeil Narimissa"],"abstract":"Large language models (LLMs) excel at general-purpose generation but often struggle with structured reasoning tasks. Recent methods like DeepSeek-R1 have shown that reinforcement learning with rule-based rewards can significantly enhance reasoning capabilities. However, reproducing such pipelines remains computationally intensive and inaccessible to most researchers. In this work, we present a modular, low-cost replication of the DeepSeek-R1 training methodology using Qwen2.5-0.5B-Instruct (a compact instruction-tuned LLM) optimized via a two-stage pipeline: Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). SFT aligns the base model to reasoning-structured prompts using LoRA-based parameter-efficient fine-tuning. GRPO then refines this policy using a critic-free reinforcement learning algorithm guided by five composable reward functions, including accuracy, reasoning presence, and formatting compliance. The entire training process was executed for under $100 USD on AWS SageMaker, demonstrating that high-impact reasoning alignment is achievable without large-scale compute. Quantitative metrics confirm strong convergence, high reward stability, and consistent output structure. This study contributes a scalable and reproducible template for aligning compact LLMs to reasoning-intensive tasks under constrained computational budgets.","url_abs":"https://www.techrxiv.org/users/922298/articles/1294225-reasoning-on-a-budget-miniaturizing-deepseek-r1-with-sft-grpo-alignment-for-instruction-tuned-llms","url_pdf":"https://d197for5662m48.cloudfront.net/documents/publicationstatus/258961/preprint_pdf/d3760eb91ad7e564a9847938ab1a4413.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reasoning-on-a-budget-miniaturizing-deepseek","repo_url":"https://github.com/EsmaeilNarimissa/aws-sft-grpo-budget-llm-finetune","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}