</> Code with Papers by Syntology

Papers › LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions

LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions

27 Apr 2023arXiv:2304.14402archive 2025-07-28

Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, Alham Fikri Aji

Large language models (LLMs) with instruction fine-tuning demonstrate superior generative capabilities. However, these models are resource-intensive. To alleviate this issue, we explore distilling knowledge from instruction-tuned LLMs into much smaller ones. To this end, we carefully develop a large set of 2.58M instructions based on both existing and newly-generated instructions. In addition to being sizable, we design our instructions to cover a broad set of topics to ensure diversity. Extensive analysis of our instruction dataset confirms its diversity, and we generate responses for these instructions using gpt-3.5-turbo. Leveraging these instructions, we fine-tune a diverse herd of models, collectively referred to as LaMini-LM, which includes models from both the encoder-decoder and decoder-only families, with varying sizes. We evaluate the performance of our models using automatic metrics on 15 different natural language processing (NLP) benchmarks, as well as through human assessment. The results demonstrate that our proposed LaMini-LM models are comparable to competitive baselines, while being much smaller in size.

PaperPDFCode

In Syntology View this paper on Syntology: its repositories, every harvested function with whether it ran, its licence and the call to fetch it.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

mbzuai-nlp/lamini-lm officialmentioned in papermentioned on GitHub report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Common Sense ReasoningCoreference ResolutionDecoderDiversityLanguage ModellingNatural Language InferenceQuestion AnsweringSentence CompletionWord Sense Disambiguation

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Common Sense Reasoning WinoGrande FLAN-T5-Large 783M Accuracy 59.9 #51 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande GPT-2-XL 1.5B Accuracy 58.3 #56 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande LaMini-F-T5 783M Accuracy 56 #60 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande LaMini-GPT 1.5B Accuracy 56 #61 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande T5-Large 738M Accuracy 55.2 #64 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande LaMini-T5 738M Accuracy 54.9 #65 of 77 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge GPT-2-XL 1.5B Accuracy 73.3 #29 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge LaMini-GPT 1.5B Accuracy 69.6 #35 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge T5-Large 738M Accuracy 66.7 #40 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge LaMini-F-T5 783M Accuracy 64.1 #44 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge LaMini-T5 738M Accuracy 59 #61 of 82 Archive leaderboard report
Natural Language Inference MultiNLI T5-Large 738M Matched 72.4 #48 of 67 Archive leaderboard report
Natural Language Inference MultiNLI T5-Large 738M Mismatched 72 #48 of 67 Archive leaderboard report
Natural Language Inference MultiNLI LaMini-GPT 1.5B Matched 67.5 #55 of 67 Archive leaderboard report
Natural Language Inference MultiNLI LaMini-GPT 1.5B Mismatched 69.3 #55 of 67 Archive leaderboard report
Natural Language Inference MultiNLI LaMini-F-T5 783M Matched 61.4 #56 of 67 Archive leaderboard report
Natural Language Inference MultiNLI LaMini-F-T5 783M Mismatched 61 #56 of 67 Archive leaderboard report
Natural Language Inference MultiNLI LaMini-T5 738M Matched 54.7 #57 of 67 Archive leaderboard report
Natural Language Inference MultiNLI LaMini-T5 738M Mismatched 55.8 #57 of 67 Archive leaderboard report
Natural Language Inference MultiNLI GPT-2-XL 1.5B Matched 36.5 #58 of 67 Archive leaderboard report
Natural Language Inference MultiNLI GPT-2-XL 1.5B Mismatched 37 #58 of 67 Archive leaderboard report
Natural Language Inference RTE T5-Large 738M Accuracy 87.4% #20 of 90 Archive leaderboard report
Natural Language Inference RTE LaMini-GPT 1.5B Accuracy 67.9% #61 of 90 Archive leaderboard report
Natural Language Inference RTE LaMini-F-T5 783M Accuracy 65% #65 of 90 Archive leaderboard report
Natural Language Inference RTE LaMini-T5 738M Accuracy 57% #81 of 90 Archive leaderboard report
Natural Language Inference RTE GPT-2-XL 1.5B Accuracy 52.3% #89 of 90 Archive leaderboard report
Question Answering OpenBookQA LaMini-GPT 1.5B Accuracy 39.8 #39 of 45 Archive leaderboard report
Question Answering OpenBookQA LaMini-T5 738M Accuracy 36 #40 of 45 Archive leaderboard report
Question Answering OpenBookQA LaMini-F-T5 783M Accuracy 34 #41 of 45 Archive leaderboard report
Question Answering OpenBookQA T5-Large 738M Accuracy 32.8 #42 of 45 Archive leaderboard report
Question Answering OpenBookQA GPT-2-XL 1.5B Accuracy 32 #43 of 45 Archive leaderboard report
Question Answering OpenBookQA FLAN-T5-Large 783M Accuracy 31.2 #44 of 45 Archive leaderboard report
Question Answering PIQA FLAN-T5-Large 783M Accuracy 72.2 #53 of 67 Archive leaderboard report
Question Answering PIQA LaMini-GPT 1.5B Accuracy 71.3 #54 of 67 Archive leaderboard report
Question Answering PIQA LaMini-F-T5 783M Accuracy 70.6 #55 of 67 Archive leaderboard report
Question Answering PIQA GPT-2-XL 1.5B Accuracy 70.5 #56 of 67 Archive leaderboard report
Question Answering PIQA LaMini-T5 738M Accuracy 67.2 #60 of 67 Archive leaderboard report
Question Answering PIQA T5-Large 738M Accuracy 55.9 #65 of 67 Archive leaderboard report
Sentence Completion HellaSwag GPT-2-XL 1.5B Accuracy 50.9 #66 of 89 Archive leaderboard report
Sentence Completion HellaSwag FLAN-T5-Large 783M Accuracy 48.7 #69 of 89 Archive leaderboard report
Sentence Completion HellaSwag LaMini-GPT 1.5B Accuracy 48.3 #70 of 89 Archive leaderboard report
Sentence Completion HellaSwag LaMini-F-T5 783M Accuracy 43.7 #72 of 89 Archive leaderboard report
Sentence Completion HellaSwag LaMini-T5 738M Accuracy 40.6 #76 of 89 Archive leaderboard report
Sentence Completion HellaSwag T5-Large 738M Accuracy 38.9 #78 of 89 Archive leaderboard report
Word Sense Disambiguation Words in Context FLAN-T5-Large 783M Accuracy 64.7 #15 of 37 Archive leaderboard report
Word Sense Disambiguation Words in Context LaMini-F-T5 783M Accuracy 63.8 #16 of 37 Archive leaderboard report
Word Sense Disambiguation Words in Context LaMini-GPT 1.5B Accuracy 52.4 #26 of 37 Archive leaderboard report
Word Sense Disambiguation Words in Context LaMini-T5 738M Accuracy 50.5 #32 of 37 Archive leaderboard report
Word Sense Disambiguation Words in Context GPT-2-XL 1.5B Accuracy 49.8 #34 of 37 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections