{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/geoeval-benchmark-for-evaluating-llms-and","title":"GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving","arxiv_id":"2402.10104","date":"2024-02-15","proceeding":null,"authors":["Jiaxin Zhang","Zhongzhi Li","Mingliang Zhang","Fei Yin","ChengLin Liu","Yashar Moshfeghi"],"abstract":"Recent advancements in large language models (LLMs) and multi-modal models (MMs) have demonstrated their remarkable capabilities in problem-solving. Yet, their proficiency in tackling geometry math problems, which necessitates an integrated understanding of both textual and visual information, has not been thoroughly evaluated. To address this gap, we introduce the GeoEval benchmark, a comprehensive collection that includes a main subset of 2,000 problems, a 750 problems subset focusing on backward reasoning, an augmented subset of 2,000 problems, and a hard subset of 300 problems. This benchmark facilitates a deeper investigation into the performance of LLMs and MMs in solving geometry math problems. Our evaluation of ten LLMs and MMs across these varied subsets reveals that the WizardMath model excels, achieving a 55.67\\% accuracy rate on the main subset but only a 6.00\\% accuracy on the hard subset. This highlights the critical need for testing models against datasets on which they have not been pre-trained. Additionally, our findings indicate that GPT-series models perform more effectively on problems they have rephrased, suggesting a promising method for enhancing model capabilities.","url_abs":"https://arxiv.org/abs/2402.10104v2","url_pdf":"https://arxiv.org/pdf/2402.10104v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"geoeval-benchmark-for-evaluating-llms-and","repo_url":"https://github.com/geoeval/geoeval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"geometry-problem-solving","task_name":"Geometry Problem Solving"},{"task_slug":"math","task_name":"Math"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.10104","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.10104"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/geoeval/geoeval","reach":{"status":"ok"}}],"summary":{"ran":13,"unverified":1},"by_repo_kind":{"official":{"samples":14,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"2c31828b054e7911","entry":"create_model_tokenizer","repo":"geoeval/geoeval","repo_kind":"official","path":"model/casual_lm_model.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/model/casual_lm_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c31828b054e7911"}},{"code_sha256_prefix":"a878f818de85fff2","entry":"create_test_prompt","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/extract_result.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/extract_result.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a878f818de85fff2"}},{"code_sha256_prefix":"cae99081e59e945b","entry":"get_chat_reponse","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/gpt_tool.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/gpt_tool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cae99081e59e945b"}},{"code_sha256_prefix":"ac99d11c002a8d64","entry":"get_opensource_llm_reponse","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/opensource_llm_tool.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/opensource_llm_tool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac99d11c002a8d64"}},{"code_sha256_prefix":"2f637a3625251af4","entry":"load_config","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/tokenized_data.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/tokenized_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f637a3625251af4"}},{"code_sha256_prefix":"1c70706ba104f6dc","entry":"load_tokenizer","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/tokenized_data.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/tokenized_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c70706ba104f6dc"}},{"code_sha256_prefix":"783a30a995ce9abd","entry":"naive_merge","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/merge_key_val.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/merge_key_val.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"783a30a995ce9abd"}},{"code_sha256_prefix":"d7dfb26704c14041","entry":"normalize_extracted_answer","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/caculate_score.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/caculate_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d7dfb26704c14041"}},{"code_sha256_prefix":"7a0e005644efec66","entry":"preprocess","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/tokenized_data.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/tokenized_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a0e005644efec66"}},{"code_sha256_prefix":"ee50a40e401c2b10","entry":"read_json","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/merge_data.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/merge_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee50a40e401c2b10"}},{"code_sha256_prefix":"6f879add2a544e52","entry":"read_pickle","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/util.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f879add2a544e52"}},{"code_sha256_prefix":"8b86d2b7f8a9bea3","entry":"safe_equal","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/caculate_score.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/caculate_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b86d2b7f8a9bea3"}},{"code_sha256_prefix":"312ea7d7bf158032","entry":"trans_choice_answer_into_string","repo":"geoeval/geoeval","repo_kind":"official","path":"tool/caculate_score.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/tool/caculate_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"312ea7d7bf158032"}},{"code_sha256_prefix":"e81f5db5843598e6","entry":"create_casuallm_model_tokenizer","repo":"geoeval/geoeval","repo_kind":"official","path":"model/casual_lm_model.py","file_url":"https://github.com/geoeval/geoeval/blob/HEAD/model/casual_lm_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e81f5db5843598e6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}