{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rulearena-a-benchmark-for-rule-guided","title":"RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios","arxiv_id":"2412.08972","date":"2024-12-12","proceeding":null,"authors":["Ruiwen Zhou","Wenyue Hua","Liangming Pan","Sitao Cheng","Xiaobao Wu","En Yu","William Yang Wang"],"abstract":"This paper introduces RuleArena, a novel and challenging benchmark designed to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning. Covering three practical domains -- airline baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs' proficiency in handling intricate natural language instructions that demand long-context understanding, logical reasoning, and accurate mathematical computation. Two key attributes distinguish RuleArena from traditional rule-based reasoning benchmarks: (1) it extends beyond standard first-order logic representations, and (2) it is grounded in authentic, practical scenarios, providing insights into the suitability and reliability of LLMs for real-world applications. Our findings reveal several notable limitations in LLMs: (1) they struggle to identify and apply the appropriate rules, frequently becoming confused by similar but distinct regulations, (2) they cannot consistently perform accurate mathematical computations, even when they correctly identify the relevant rules, and (3) in general, they perform poorly in the benchmark. These results highlight significant challenges in advancing LLMs' rule-guided reasoning capabilities in real-life applications.","url_abs":"https://arxiv.org/abs/2412.08972v1","url_pdf":"https://arxiv.org/pdf/2412.08972v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rulearena-a-benchmark-for-rule-guided","repo_url":"https://github.com/skyriver-2000/rulearena","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"},{"task_slug":"long-context-understanding","task_name":"Long-Context Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2412.08972","atlas_url":"https://app.syntology.ai/?focus=2412.08972","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.08972"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/skyriver-2000/rulearena","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"33729edf0f537f04","entry":"build_query_prompt","repo":"skyriver-2000/rulearena","repo_kind":"official","path":"nba/micro_evaluation.py","file_url":"https://github.com/skyriver-2000/rulearena/blob/HEAD/nba/micro_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33729edf0f537f04"}},{"code_sha256_prefix":"6672f83bb9ba6c7b","entry":"gen_question","repo":"skyriver-2000/rulearena","repo_kind":"official","path":"airline/gen_questions.py","file_url":"https://github.com/skyriver-2000/rulearena/blob/HEAD/airline/gen_questions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6672f83bb9ba6c7b"}},{"code_sha256_prefix":"27015c02d456d414","entry":"generate_sche_1","repo":"skyriver-2000/rulearena","repo_kind":"official","path":"tax/gen_payer.py","file_url":"https://github.com/skyriver-2000/rulearena/blob/HEAD/tax/gen_payer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27015c02d456d414"}},{"code_sha256_prefix":"9ddfec7ba3d407f2","entry":"generate_sche_3","repo":"skyriver-2000/rulearena","repo_kind":"official","path":"tax/gen_payer.py","file_url":"https://github.com/skyriver-2000/rulearena/blob/HEAD/tax/gen_payer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ddfec7ba3d407f2"}},{"code_sha256_prefix":"34570a699cf7b596","entry":"generate_sche_c","repo":"skyriver-2000/rulearena","repo_kind":"official","path":"tax/gen_payer.py","file_url":"https://github.com/skyriver-2000/rulearena/blob/HEAD/tax/gen_payer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34570a699cf7b596"}},{"code_sha256_prefix":"dde255079fb4a398","entry":"gpt","repo":"skyriver-2000/rulearena","repo_kind":"official","path":"nba/micro_evaluation.py","file_url":"https://github.com/skyriver-2000/rulearena/blob/HEAD/nba/micro_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dde255079fb4a398"}},{"code_sha256_prefix":"1dada379afb0981c","entry":"invert_order","repo":"skyriver-2000/rulearena","repo_kind":"official","path":"airline/compute_answer.py","file_url":"https://github.com/skyriver-2000/rulearena/blob/HEAD/airline/compute_answer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1dada379afb0981c"}},{"code_sha256_prefix":"18fc43eb30bd48af","entry":"load_problems","repo":"skyriver-2000/rulearena","repo_kind":"official","path":"airline/micro_evaluation.py","file_url":"https://github.com/skyriver-2000/rulearena/blob/HEAD/airline/micro_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18fc43eb30bd48af"}},{"code_sha256_prefix":"b77c07cf7776a152","entry":"parse_rule_application","repo":"skyriver-2000/rulearena","repo_kind":"official","path":"nba/micro_evaluation.py","file_url":"https://github.com/skyriver-2000/rulearena/blob/HEAD/nba/micro_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b77c07cf7776a152"}},{"code_sha256_prefix":"0001e5e0d6c0e6b5","entry":"standard_deduction","repo":"skyriver-2000/rulearena","repo_kind":"official","path":"tax/micro_evaluation.py","file_url":"https://github.com/skyriver-2000/rulearena/blob/HEAD/tax/micro_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0001e5e0d6c0e6b5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}