{"url":"/dataset/numglue","name":"NumGLUE","full_name":null,"description_markdown":"The **NumGLUE dataset** is a valuable resource developed by the **Allen Institute for AI**. It focuses on evaluating the performance of AI systems in **mathematical reasoning tasks** that involve numbers within natural language text. Here are the key details about NumGLUE:\r\n\r\n1. **Purpose and Inspiration**:\r\n   - Drawing inspiration from the **GLUE benchmark**, which was designed for natural language understanding, NumGLUE aims to assess AI systems' ability to reason with numbers.\r\n   - Unlike GLUE, which covers a wide range of NLP tasks, NumGLUE specifically targets tasks that require **simple arithmetic understanding**.\r\n\r\n2. **Tasks**:\r\n   - NumGLUE consists of **eight different tasks**, each involving numerical reasoning:\r\n     - **Commonsense + Arithmetic Reasoning**\r\n     - **Domain Specific + Arithmetic Reasoning**\r\n     - **Commonsense + Quantitative Comparison**\r\n     - **Fill-in-the-blanks Format**\r\n     - **Reading Comprehension (RC) + Explicit Numerical Reasoning**\r\n     - **Reading Comprehension (RC) + Implicit Numerical Reasoning**\r\n\r\n3. **Challenges and Performance**:\r\n   - Despite the availability of neural models, including state-of-the-art large-scale language models, **NumGLUE remains unsolved**.\r\n   - These models perform significantly worse than humans, with an average gap of **46.4%**.\r\n   - The dataset encourages knowledge sharing across tasks, especially for those with limited training data. Joint training on all tasks yields superior performance.\r\n\r\n4. **Importance**:\r\n   - NumGLUE promotes the development of systems capable of robust and general **arithmetic reasoning within language**.\r\n   - It serves as a stepping stone toward more complex mathematical reasoning.\r\n\r\n(1) NumGLUE Dataset — Allen Institute for AI. https://allenai.org/data/numglue.\r\n(2) GitHub - allenai/numglue: NumGLUE: A Suite of Fundamental yet .... https://github.com/allenai/numglue.\r\n(3) nyu-mll/glue · Datasets at Hugging Face. https://huggingface.co/datasets/nyu-mll/glue.","description_withheld":null,"homepage":"https://allenai.org/data/numglue","introduced_date":"2022-04-12","introduced_date_note":null,"introduced_by":{"paper":"/paper/numglue-a-suite-of-fundamental-yet","title":"NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks","first_author":"Swaroop Mishra","url":null},"license":null,"modalities":[],"tasks":[],"languages":[],"variants":["NumGLUE"],"data_loaders":[],"num_papers_in_archive":23,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}