{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/are-large-language-models-good-statisticians","title":"Are Large Language Models Good Statisticians?","arxiv_id":"2406.07815","date":"2024-06-12","proceeding":null,"authors":["Yizhang Zhu","Shiyin Du","Boyan Li","Yuyu Luo","Nan Tang"],"abstract":"Large Language Models (LLMs) have demonstrated impressive capabilities across a range of scientific tasks including mathematics, physics, and chemistry. Despite their successes, the effectiveness of LLMs in handling complex statistical tasks remains systematically under-explored. To bridge this gap, we introduce StatQA, a new benchmark designed for statistical analysis tasks. StatQA comprises 11,623 examples tailored to evaluate LLMs' proficiency in specialized statistical tasks and their applicability assessment capabilities, particularly for hypothesis testing methods. We systematically experiment with representative LLMs using various prompting strategies and show that even state-of-the-art models such as GPT-4o achieve a best performance of only 64.83%, indicating significant room for improvement. Notably, while open-source LLMs (e.g. LLaMA-3) show limited capability, those fine-tuned ones exhibit marked improvements, outperforming all in-context learning-based methods (e.g. GPT-4o). Moreover, our comparative human experiments highlight a striking contrast in error types between LLMs and humans: LLMs primarily make applicability errors, whereas humans mostly make statistical task confusion errors. This divergence highlights distinct areas of proficiency and deficiency, suggesting that combining LLM and human expertise could lead to complementary strengths, inviting further investigation into their collaborative potential. Our source code and data are available at https://statqa.github.io/.","url_abs":"https://arxiv.org/abs/2406.07815v2","url_pdf":"https://arxiv.org/pdf/2406.07815v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"are-large-language-models-good-statisticians","repo_url":"https://github.com/derrickzhuyz/statqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.07815","atlas_url":"https://app.syntology.ai/?focus=2406.07815","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07815"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/derrickzhuyz/statqa","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran":8},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"0f221deb0d99d0ac","entry":"adjust_difficulty_for_benchmark","repo":"derrickzhuyz/statqa","repo_kind":"official","path":"Construction/benchmark_balancing.py","file_url":"https://github.com/derrickzhuyz/statqa/blob/HEAD/Construction/benchmark_balancing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0f221deb0d99d0ac"}},{"code_sha256_prefix":"0d6eb666267a07a1","entry":"compare_and_count_answer_row","repo":"derrickzhuyz/statqa","repo_kind":"official","path":"analyze_model_answer.py","file_url":"https://github.com/derrickzhuyz/statqa/blob/HEAD/analyze_model_answer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0d6eb666267a07a1"}},{"code_sha256_prefix":"25b96370b6e5e212","entry":"filter_not_applicable","repo":"derrickzhuyz/statqa","repo_kind":"official","path":"Construction/benchmark_postprocessing.py","file_url":"https://github.com/derrickzhuyz/statqa/blob/HEAD/Construction/benchmark_postprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"25b96370b6e5e212"}},{"code_sha256_prefix":"210bb3d7d71304d8","entry":"is_categorical","repo":"derrickzhuyz/statqa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/derrickzhuyz/statqa/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"210bb3d7d71304d8"}},{"code_sha256_prefix":"d46d776b53eab8dd","entry":"is_quantitative","repo":"derrickzhuyz/statqa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/derrickzhuyz/statqa/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d46d776b53eab8dd"}},{"code_sha256_prefix":"1415fc951e81f32d","entry":"mock_paraphrase_question_gpt","repo":"derrickzhuyz/statqa","repo_kind":"official","path":"Construction/benchmark_balancing.py","file_url":"https://github.com/derrickzhuyz/statqa/blob/HEAD/Construction/benchmark_balancing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1415fc951e81f32d"}},{"code_sha256_prefix":"39716b18bfa7f20e","entry":"mock_refine_question_gpt","repo":"derrickzhuyz/statqa","repo_kind":"official","path":"Construction/gpt_refine_question.py","file_url":"https://github.com/derrickzhuyz/statqa/blob/HEAD/Construction/gpt_refine_question.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"39716b18bfa7f20e"}},{"code_sha256_prefix":"8a8442ec614a7b93","entry":"model_answer_score_policy","repo":"derrickzhuyz/statqa","repo_kind":"official","path":"analyze_model_answer.py","file_url":"https://github.com/derrickzhuyz/statqa/blob/HEAD/analyze_model_answer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8a8442ec614a7b93"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}