{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fermat-an-alternative-to-accuracy-for","title":"FERMAT: An Alternative to Accuracy for Numerical Reasoning","arxiv_id":"2305.17491","date":"2023-05-27","proceeding":null,"authors":["Jasivan Alex Sivakumar","Nafise Sadat Moosavi"],"abstract":"While pre-trained language models achieve impressive performance on various NLP benchmarks, they still struggle with tasks that require numerical reasoning. Recent advances in improving numerical reasoning are mostly achieved using very large language models that contain billions of parameters and are not accessible to everyone. In addition, numerical reasoning is measured using a single score on existing datasets. As a result, we do not have a clear understanding of the strengths and shortcomings of existing models on different numerical reasoning aspects and therefore, potential ways to improve them apart from scaling them up. Inspired by CheckList (Ribeiro et al., 2020), we introduce a multi-view evaluation set for numerical reasoning in English, called FERMAT. Instead of reporting a single score on a whole dataset, FERMAT evaluates models on various key numerical reasoning aspects such as number understanding, mathematical operations, and training dependency. Apart from providing a comprehensive evaluation of models on different numerical reasoning aspects, FERMAT enables a systematic and automated generation of an arbitrarily large training or evaluation set for each aspect.The datasets and codes are publicly available to generate further multi-view data for ulterior tasks and languages.","url_abs":"https://arxiv.org/abs/2305.17491v1","url_pdf":"https://arxiv.org/pdf/2305.17491v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fermat-an-alternative-to-accuracy-for","repo_url":"https://github.com/jasivan/fermat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.17491","atlas_url":"https://app.syntology.ai/?focus=2305.17491","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.17491"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jasivan/FERMAT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b46eeaf6dec1e556","entry":"char_level_representation_encoding","repo":"jasivan/FERMAT","repo_kind":"official","path":"codes/char_level_rep.py","file_url":"https://github.com/jasivan/FERMAT/blob/HEAD/codes/char_level_rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b46eeaf6dec1e556"}},{"code_sha256_prefix":"4bdcc6fd40a2753c","entry":"char_level_representation_impl","repo":"jasivan/FERMAT","repo_kind":"official","path":"codes/char_level_rep.py","file_url":"https://github.com/jasivan/FERMAT/blob/HEAD/codes/char_level_rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4bdcc6fd40a2753c"}},{"code_sha256_prefix":"d927911a7d756812","entry":"compute_metrics","repo":"jasivan/FERMAT","repo_kind":"official","path":"codes/evaluate_from_ckpt.py","file_url":"https://github.com/jasivan/FERMAT/blob/HEAD/codes/evaluate_from_ckpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d927911a7d756812"}},{"code_sha256_prefix":"e5d104e749b6caaa","entry":"compute_metrics","repo":"jasivan/FERMAT","repo_kind":"official","path":"codes/evaluate_training.py","file_url":"https://github.com/jasivan/FERMAT/blob/HEAD/codes/evaluate_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5d104e749b6caaa"}},{"code_sha256_prefix":"bd8cd57d2905efde","entry":"pad_or_trim","repo":"jasivan/FERMAT","repo_kind":"official","path":"codes/char_level_rep.py","file_url":"https://github.com/jasivan/FERMAT/blob/HEAD/codes/char_level_rep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd8cd57d2905efde"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}