{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/assessing-the-creativity-of-llms-in-proposing","title":"Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems","arxiv_id":"2410.18336","date":"2024-10-24","proceeding":null,"authors":["Junyi Ye","Jingyi Gu","Xinyun Zhao","Wenpeng Yin","Guiling Wang"],"abstract":"The mathematical capabilities of AI systems are complex and multifaceted. Most existing research has predominantly focused on the correctness of AI-generated solutions to mathematical problems. In this work, we argue that beyond producing correct answers, AI systems should also be capable of, or assist humans in, developing novel solutions to mathematical challenges. This study explores the creative potential of Large Language Models (LLMs) in mathematical reasoning, an aspect that has received limited attention in prior research. We introduce a novel framework and benchmark, CreativeMath, which encompasses problems ranging from middle school curricula to Olympic-level competitions, designed to assess LLMs' ability to propose innovative solutions after some known solutions have been provided. Our experiments demonstrate that, while LLMs perform well on standard mathematical tasks, their capacity for creative problem-solving varies considerably. Notably, the Gemini-1.5-Pro model outperformed other LLMs in generating novel solutions. This research opens a new frontier in evaluating AI creativity, shedding light on both the strengths and limitations of LLMs in fostering mathematical innovation, and setting the stage for future developments in AI-assisted mathematical discovery.","url_abs":"https://arxiv.org/abs/2410.18336v1","url_pdf":"https://arxiv.org/pdf/2410.18336v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"assessing-the-creativity-of-llms-in-proposing","repo_url":"https://github.com/junyiye/creativemath","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.18336","atlas_url":"https://app.syntology.ai/?focus=2410.18336","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.18336"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/junyiye/creativemath","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"67f0150cdc2e0c55","entry":"extract_yes_no","repo":"junyiye/creativemath","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/junyiye/creativemath/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67f0150cdc2e0c55"}},{"code_sha256_prefix":"1ecddc7b2d1fd799","entry":"load_coarse_grained_novelty_evaluation_prompt","repo":"junyiye/creativemath","repo_kind":"official","path":"src/prompts/prompts.py","file_url":"https://github.com/junyiye/creativemath/blob/HEAD/src/prompts/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ecddc7b2d1fd799"}},{"code_sha256_prefix":"0c3462a9fe6960df","entry":"load_correctness_evaluation_prompt","repo":"junyiye/creativemath","repo_kind":"official","path":"src/prompts/prompts.py","file_url":"https://github.com/junyiye/creativemath/blob/HEAD/src/prompts/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c3462a9fe6960df"}},{"code_sha256_prefix":"f643f9552ec644d5","entry":"load_json","repo":"junyiye/creativemath","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/junyiye/creativemath/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f643f9552ec644d5"}},{"code_sha256_prefix":"3521b0e9e57fc1a2","entry":"load_messages","repo":"junyiye/creativemath","repo_kind":"official","path":"src/models/prompt_utils.py","file_url":"https://github.com/junyiye/creativemath/blob/HEAD/src/models/prompt_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3521b0e9e57fc1a2"}},{"code_sha256_prefix":"5f3162b236b7c2da","entry":"load_novel_solution_generation_prompt","repo":"junyiye/creativemath","repo_kind":"official","path":"src/prompts/prompts.py","file_url":"https://github.com/junyiye/creativemath/blob/HEAD/src/prompts/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f3162b236b7c2da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}