{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-independent-evaluation-of-chatgpt-on","title":"An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)","arxiv_id":"2302.13814","date":"2023-02-23","proceeding":null,"authors":["Paulo Shakarian","Abhinav Koyyalamudi","Noel Ngu","Lakshmivihari Mareedu"],"abstract":"We study the performance of a commercially available large language model (LLM) known as ChatGPT on math word problems (MWPs) from the dataset DRAW-1K. To our knowledge, this is the first independent evaluation of ChatGPT. We found that ChatGPT's performance changes dramatically based on the requirement to show its work, failing 20% of the time when it provides work compared with 84% when it does not. Further several factors about MWPs relating to the number of unknowns and number of operations that lead to a higher probability of failure when compared with the prior, specifically noting (across all experiments) that the probability of failure increases linearly with the number of addition and subtraction operations. We also have released the dataset of ChatGPT's responses to the MWPs to support further work on the characterization of LLM performance and present baseline machine learning models to predict if ChatGPT can correctly answer an MWP. We have released a dataset comprised of ChatGPT's responses to support further research in this area.","url_abs":"https://arxiv.org/abs/2302.13814v2","url_pdf":"https://arxiv.org/pdf/2302.13814v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-independent-evaluation-of-chatgpt-on","repo_url":"https://github.com/lab-v2/chatgpt_mwp_eval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"CC0-1.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"math","task_name":"Math"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2302.13814","atlas_url":"https://app.syntology.ai/?focus=2302.13814","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.13814"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lab-v2/chatgpt_mwp_eval","reach":{"status":"ok","spdx":"CC0-1.0"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5e31e54789ef21f4","entry":"Scan_Answers","repo":"lab-v2/chatgpt_mwp_eval","repo_kind":"official","path":"data_with_implementation/source/JSONGPT.py","file_url":"https://github.com/lab-v2/chatgpt_mwp_eval/blob/HEAD/data_with_implementation/source/JSONGPT.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5e31e54789ef21f4"}},{"code_sha256_prefix":"e8c0ea9158bf8877","entry":"Scan_Answers","repo":"lab-v2/chatgpt_mwp_eval","repo_kind":"official","path":"data_with_implementation/source/trailGPT.py","file_url":"https://github.com/lab-v2/chatgpt_mwp_eval/blob/HEAD/data_with_implementation/source/trailGPT.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8c0ea9158bf8877"}},{"code_sha256_prefix":"7392bcd76e77496b","entry":"compare_answers","repo":"lab-v2/chatgpt_mwp_eval","repo_kind":"official","path":"data_with_implementation/source/JSONGPT.py","file_url":"https://github.com/lab-v2/chatgpt_mwp_eval/blob/HEAD/data_with_implementation/source/JSONGPT.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7392bcd76e77496b"}},{"code_sha256_prefix":"8c10edea24b263eb","entry":"find_Ans","repo":"lab-v2/chatgpt_mwp_eval","repo_kind":"official","path":"data_with_implementation/source/JSONGPT.py","file_url":"https://github.com/lab-v2/chatgpt_mwp_eval/blob/HEAD/data_with_implementation/source/JSONGPT.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c10edea24b263eb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}