Papers › Lila: A Unified Benchmark for Mathematical Reasoning
Lila: A Unified Benchmark for Mathematical Reasoning
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, Ashwin Kalyan
Mathematical reasoning skills are essential for general-purpose intelligent systems to perform tasks from grocery shopping to climate modeling. Towards evaluating and improving AI systems in this domain, we propose LILA, a unified mathematical reasoning benchmark consisting of 23 diverse tasks along four dimensions: (i) mathematical abilities e.g., arithmetic, calculus (ii) language format e.g., question-answering, fill-in-the-blanks (iii) language diversity e.g., no language, simple language (iv) external knowledge e.g., commonsense, physics. We construct our benchmark by extending 20 datasets benchmark by collecting task instructions and solutions in the form of Python programs, thereby obtaining explainable solutions in addition to the correct answer. We additionally introduce two evaluation datasets to measure out-of-distribution performance and robustness to language perturbation. Finally, we introduce BHASKARA, a general-purpose mathematical reasoning model trained on LILA. Importantly, we find that multi-tasking leads to significant improvements (average relative improvement of 21.83% F1 score vs. single-task models), while the best performing model only obtains 60.40%, indicating the room for improvement in general mathematical reasoning and understanding.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Mathematical Reasoning | Lila (IID) | Codex (Few-Shot, 175B) | Accuracy | 0.604 | #1 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (IID) | Bhāskara-P (Fine-tuned, 2.7B) | Accuracy | 0.48 | #2 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (IID) | Neo-P (Fine-tuned, 2.7B) | Accuracy | 0.394 | #3 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (IID) | GPT-3 (Few-Shot, 175B) | Accuracy | 0.384 | #4 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (IID) | Bhāskara-A (Fine-tuned, 2.7B) | Accuracy | 0.252 | #5 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (IID) | Neo-A (Fine-tuned, 2.7B) | Accuracy | 0.204 | #6 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (OOD) | Codex (Few-Shot, 175B) | Accuracy | 0.586 | #1 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (OOD) | Bhāskara-P (Fine-tuned, 2.7B) | Accuracy | 0.448 | #2 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (OOD) | GPT-3 (Few-Shot, 175B) | Accuracy | 0.384 | #3 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (OOD) | Bhāskara-A (Fine-tuned, 2.7B) | Accuracy | 0.268 | #4 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (OOD) | Neo-P (Fine-tuned, 2.7B) | Accuracy | 0.238 | #5 of 6 | Archive leaderboard | report |
| Mathematical Reasoning | Lila (OOD) | Neo-A (Fine-tuned, 2.7B) | Accuracy | 0.177 | #6 of 6 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections