Papers › Distilling Reasoning Capabilities into Smaller Language Models

Distilling Reasoning Capabilities into Smaller Language Models

1 Dec 2022arXiv:2212.00193archive 2025-07-28

Kumar Shridhar, Alessandro Stolfo, Mrinmaya Sachan

Step-by-step reasoning approaches like chain of thought (CoT) have proved to be very effective in inducing reasoning capabilities in large language models. However, the success of the CoT approach is fundamentally tied to the model size, and billion parameter-scale models are often needed to get CoT to work. In this paper, we propose a knowledge distillation approach that leverages the step-by-step CoT reasoning capabilities of larger models and distills these abilities into smaller models. In this work, we propose an alternative reasoning scheme, Socratic CoT, that learns a decomposition of the original problem into a sequence of subproblems and uses it to guide the intermediate reasoning steps. We use Socratic CoT to train a combination of two small distilled models: a problem decomposer and a subproblem solver. In practice, given a new problem, the two distilled models work in sync to decompose and solve complex problems. On multiple reasoning datasets (GSM8K, StrategyQA, and SVAMP), our proposed distillation strategies boosts the performance of smaller models over 70% compared to the baselines. Finally, we investigate when Socratic CoT is an effective alternative to CoT, demonstrating cases where a much smaller model (GPT-2 large) can outperform a 10X larger model (GPT-3 6B). Our code is available here: https://github.com/kumar-shridhar/Distiiling-LM

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

kumar-shridhar/distiiling-lm officialmentioned in paper report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

GSM8K Knowledge Distillation Problem Decomposition StrategyQA

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Adam Attention Attention Dropout BPE Cosine Annealing Dense Connections Discriminative Fine-Tuning Dropout GPT-2 GPT-3 Knowledge Distillation Layer Normalization Linear Layer Linear Warmup With Cosine Annealing Multi-Head Attention Residual Connection Softmax Weight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections