{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mirage-evaluating-and-explaining-inductive","title":"MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models","arxiv_id":"2410.09542","date":"2024-10-12","proceeding":null,"authors":["Jiachun Li","Pengfei Cao","Zhuoran Jin","Yubo Chen","Kang Liu","Jun Zhao"],"abstract":"Inductive reasoning is an essential capability for large language models (LLMs) to achieve higher intelligence, which requires the model to generalize rules from observed facts and then apply them to unseen examples. We present {\\scshape Mirage}, a synthetic dataset that addresses the limitations of previous work, specifically the lack of comprehensive evaluation and flexible test data. In it, we evaluate LLMs' capabilities in both the inductive and deductive stages, allowing for flexible variation in input distribution, task scenario, and task difficulty to analyze the factors influencing LLMs' inductive reasoning. Based on these multi-faceted evaluations, we demonstrate that the LLM is a poor rule-based reasoner. In many cases, when conducting inductive reasoning, they do not rely on a correct rule to answer the unseen case. From the perspectives of different prompting methods, observation numbers, and task forms, models tend to consistently conduct correct deduction without correct inductive rules. Besides, we find that LLMs are good neighbor-based reasoners. In the inductive reasoning process, the model tends to focus on observed facts that are close to the current test example in feature space. By leveraging these similar examples, the model maintains strong inductive capabilities within a localized region, significantly improving its deductive performance.","url_abs":"https://arxiv.org/abs/2410.09542v1","url_pdf":"https://arxiv.org/pdf/2410.09542v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.09542","atlas_url":"https://app.syntology.ai/?focus=2410.09542","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.09542"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/BugMakerzzz/mirage","reach":null}],"summary":{"ran_fixture":5},"by_repo_kind":{"found_in_text":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"ae6d6e516578392b","entry":"add","repo":"BugMakerzzz/mirage","repo_kind":"found_in_text","path":"src/generate_data.py","file_url":"https://github.com/BugMakerzzz/mirage/blob/HEAD/src/generate_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae6d6e516578392b"}},{"code_sha256_prefix":"28c5b1bb0461619e","entry":"convert_vec_to_str","repo":"BugMakerzzz/mirage","repo_kind":"found_in_text","path":"src/generate_question.py","file_url":"https://github.com/BugMakerzzz/mirage/blob/HEAD/src/generate_question.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28c5b1bb0461619e"}},{"code_sha256_prefix":"a37c3660216eef4d","entry":"map","repo":"BugMakerzzz/mirage","repo_kind":"found_in_text","path":"src/generate_data.py","file_url":"https://github.com/BugMakerzzz/mirage/blob/HEAD/src/generate_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a37c3660216eef4d"}},{"code_sha256_prefix":"e59a5699bd1d9193","entry":"pad","repo":"BugMakerzzz/mirage","repo_kind":"found_in_text","path":"src/generate_data.py","file_url":"https://github.com/BugMakerzzz/mirage/blob/HEAD/src/generate_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e59a5699bd1d9193"}},{"code_sha256_prefix":"f080e9f62b29c033","entry":"str_operate","repo":"BugMakerzzz/mirage","repo_kind":"found_in_text","path":"src/generate_question.py","file_url":"https://github.com/BugMakerzzz/mirage/blob/HEAD/src/generate_question.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f080e9f62b29c033"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}