{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/feabench-evaluating-language-models-on","title":"FEABench: Evaluating Language Models on Multiphysics Reasoning Ability","arxiv_id":"2504.06260","date":"2025-04-08","proceeding":null,"authors":["Nayantara Mudur","HAO CUI","Subhashini Venugopalan","Paul Raccuglia","Michael P. Brenner","Peter Norgaard"],"abstract":"Building precise simulations of the real world and invoking numerical solvers to answer quantitative problems is an essential requirement in engineering and science. We present FEABench, a benchmark to evaluate the ability of large language models (LLMs) and LLM agents to simulate and solve physics, mathematics and engineering problems using finite element analysis (FEA). We introduce a comprehensive evaluation scheme to investigate the ability of LLMs to solve these problems end-to-end by reasoning over natural language problem descriptions and operating COMSOL Multiphysics$^\\circledR$, an FEA software, to compute the answers. We additionally design a language model agent equipped with the ability to interact with the software through its Application Programming Interface (API), examine its outputs and use tools to improve its solutions over multiple iterations. Our best performing strategy generates executable API calls 88% of the time. LLMs that can successfully interact with and operate FEA software to solve problems such as those in our benchmark would push the frontiers of automation in engineering. Acquiring this capability would augment LLMs' reasoning skills with the precision of numerical solvers and advance the development of autonomous systems that can tackle complex problems in the real world. The code is available at https://github.com/google/feabench","url_abs":"https://arxiv.org/abs/2504.06260v1","url_pdf":"https://arxiv.org/pdf/2504.06260v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"feabench-evaluating-language-models-on","repo_url":"https://github.com/google/feabench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.06260","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.06260"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google/feabench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":13},"by_repo_kind":{"official":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ed781a24882efe4e","entry":"call_function_with_retry","repo":"google/feabench","repo_kind":"official","path":"retry_lib.py","file_url":"https://github.com/google/feabench/blob/HEAD/retry_lib.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed781a24882efe4e"}},{"code_sha256_prefix":"3c7c5b9ae36a8255","entry":"compactify_java","repo":"google/feabench","repo_kind":"official","path":"parse_java_api.py","file_url":"https://github.com/google/feabench/blob/HEAD/parse_java_api.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c7c5b9ae36a8255"}},{"code_sha256_prefix":"9e9a06f31482e563","entry":"diff_score","repo":"google/feabench","repo_kind":"official","path":"common/eval/api_score.py","file_url":"https://github.com/google/feabench/blob/HEAD/common/eval/api_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e9a06f31482e563"}},{"code_sha256_prefix":"0890377be68a2e7c","entry":"get_default_gemini_external_model_config","repo":"google/feabench","repo_kind":"official","path":"common/llm_client_builder.py","file_url":"https://github.com/google/feabench/blob/HEAD/common/llm_client_builder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0890377be68a2e7c"}},{"code_sha256_prefix":"9c5ce7b2768db975","entry":"get_problem_description_for_task_version","repo":"google/feabench","repo_kind":"official","path":"common/prompt_generation.py","file_url":"https://github.com/google/feabench/blob/HEAD/common/prompt_generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c5ce7b2768db975"}},{"code_sha256_prefix":"608a24fdd2354dff","entry":"model_tree_diff","repo":"google/feabench","repo_kind":"official","path":"common/eval/api_score.py","file_url":"https://github.com/google/feabench/blob/HEAD/common/eval/api_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"608a24fdd2354dff"}},{"code_sha256_prefix":"4255f03f1cc60af1","entry":"parse_java","repo":"google/feabench","repo_kind":"official","path":"generate_feabench_large/parse_tutorial_code.py","file_url":"https://github.com/google/feabench/blob/HEAD/generate_feabench_large/parse_tutorial_code.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4255f03f1cc60af1"}},{"code_sha256_prefix":"b4d1ea1081fc8176","entry":"pythonize_java_api","repo":"google/feabench","repo_kind":"official","path":"parse_java_api.py","file_url":"https://github.com/google/feabench/blob/HEAD/parse_java_api.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4d1ea1081fc8176"}},{"code_sha256_prefix":"c81a30deb72f981e","entry":"replace_in_prompt","repo":"google/feabench","repo_kind":"official","path":"common/prompt_generation.py","file_url":"https://github.com/google/feabench/blob/HEAD/common/prompt_generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c81a30deb72f981e"}},{"code_sha256_prefix":"f647063765ee8260","entry":"retry_wrapper","repo":"google/feabench","repo_kind":"official","path":"common/llm_client_builder.py","file_url":"https://github.com/google/feabench/blob/HEAD/common/llm_client_builder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f647063765ee8260"}},{"code_sha256_prefix":"d8af8971c6b7c8c8","entry":"separate_run_method_body","repo":"google/feabench","repo_kind":"official","path":"parse_java_api.py","file_url":"https://github.com/google/feabench/blob/HEAD/parse_java_api.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d8af8971c6b7c8c8"}},{"code_sha256_prefix":"eddecaba92735bbe","entry":"specify_prompt_template","repo":"google/feabench","repo_kind":"official","path":"common/prompt_generation.py","file_url":"https://github.com/google/feabench/blob/HEAD/common/prompt_generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eddecaba92735bbe"}},{"code_sha256_prefix":"e05671e1181ec77d","entry":"split_string_by_delimiter","repo":"google/feabench","repo_kind":"official","path":"common_parsing_utils.py","file_url":"https://github.com/google/feabench/blob/HEAD/common_parsing_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e05671e1181ec77d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}