{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/seemingly-plausible-distractors-in-multi-hop","title":"Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?","arxiv_id":"2409.05197","date":"2024-09-08","proceeding":null,"authors":["Neeladri Bhuiya","Viktor Schlegel","Stefan Winkler"],"abstract":"State-of-the-art Large Language Models (LLMs) are accredited with an increasing number of different capabilities, ranging from reading comprehension, over advanced mathematical and reasoning skills to possessing scientific knowledge. In this paper we focus on their multi-hop reasoning capability: the ability to identify and integrate information from multiple textual sources. Given the concerns with the presence of simplifying cues in existing multi-hop reasoning benchmarks, which allow models to circumvent the reasoning requirement, we set out to investigate, whether LLMs are prone to exploiting such simplifying cues. We find evidence that they indeed circumvent the requirement to perform multi-hop reasoning, but they do so in more subtle ways than what was reported about their fine-tuned pre-trained language model (PLM) predecessors. Motivated by this finding, we propose a challenging multi-hop reasoning benchmark, by generating seemingly plausible multi-hop reasoning chains, which ultimately lead to incorrect answers. We evaluate multiple open and proprietary state-of-the-art LLMs, and find that their performance to perform multi-hop reasoning is affected, as indicated by up to 45% relative decrease in F1 score when presented with such seemingly plausible alternatives. We conduct a deeper analysis and find evidence that while LLMs tend to ignore misleading lexical cues, misleading reasoning paths indeed present a significant challenge.","url_abs":"https://arxiv.org/abs/2409.05197v3","url_pdf":"https://arxiv.org/pdf/2409.05197v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"seemingly-plausible-distractors-in-multi-hop","repo_url":"https://github.com/zawedcvg/are-large-language-models-attentive-readers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.05197","atlas_url":"https://app.syntology.ai/?focus=2409.05197","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.05197"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zawedcvg/are-large-language-models-attentive-readers","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/zawedcvg/Are-Large-Language-Models-Attentive-Readers","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_violates":1,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dcb601348781165b","entry":"backoff_delay","repo":"zawedcvg/Are-Large-Language-Models-Attentive-Readers","repo_kind":"official","path":"create_fake_para.py","file_url":"https://github.com/zawedcvg/Are-Large-Language-Models-Attentive-Readers/blob/HEAD/create_fake_para.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dcb601348781165b"}},{"code_sha256_prefix":"52795004151377fd","entry":"chain_of_thought_prompt","repo":"zawedcvg/Are-Large-Language-Models-Attentive-Readers","repo_kind":"official","path":"chain_of_thought_prompts.py","file_url":"https://github.com/zawedcvg/Are-Large-Language-Models-Attentive-Readers/blob/HEAD/chain_of_thought_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52795004151377fd"}},{"code_sha256_prefix":"f340c851521bcbc1","entry":"chain_of_thought_prompt_control","repo":"zawedcvg/Are-Large-Language-Models-Attentive-Readers","repo_kind":"official","path":"chain_of_thought_prompts.py","file_url":"https://github.com/zawedcvg/Are-Large-Language-Models-Attentive-Readers/blob/HEAD/chain_of_thought_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f340c851521bcbc1"}},{"code_sha256_prefix":"7e921f20bf4f544e","entry":"chain_of_thought_prompt_subqa","repo":"zawedcvg/Are-Large-Language-Models-Attentive-Readers","repo_kind":"official","path":"chain_of_thought_prompts.py","file_url":"https://github.com/zawedcvg/Are-Large-Language-Models-Attentive-Readers/blob/HEAD/chain_of_thought_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e921f20bf4f544e"}},{"code_sha256_prefix":"108b8b142bf27098","entry":"create_context","repo":"zawedcvg/Are-Large-Language-Models-Attentive-Readers","repo_kind":"official","path":"create_prompts.py","file_url":"https://github.com/zawedcvg/Are-Large-Language-Models-Attentive-Readers/blob/HEAD/create_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"108b8b142bf27098"}},{"code_sha256_prefix":"b7e16417bab14929","entry":"create_fake_para","repo":"zawedcvg/Are-Large-Language-Models-Attentive-Readers","repo_kind":"official","path":"create_prompts.py","file_url":"https://github.com/zawedcvg/Are-Large-Language-Models-Attentive-Readers/blob/HEAD/create_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7e16417bab14929"}},{"code_sha256_prefix":"725fa390ba83e35b","entry":"is_enough_paragraphs","repo":"zawedcvg/are-large-language-models-attentive-readers","repo_kind":"official","path":"create_prompts_with_fake_para.py","file_url":"https://github.com/zawedcvg/are-large-language-models-attentive-readers/blob/HEAD/create_prompts_with_fake_para.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"725fa390ba83e35b"}},{"code_sha256_prefix":"23d10d95d360854a","entry":"is_in","repo":"zawedcvg/Are-Large-Language-Models-Attentive-Readers","repo_kind":"official","path":"create_fake_question_tuples.py","file_url":"https://github.com/zawedcvg/Are-Large-Language-Models-Attentive-Readers/blob/HEAD/create_fake_question_tuples.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23d10d95d360854a"}},{"code_sha256_prefix":"187e2e2e5060ca39","entry":"create_dictionary_same_index","repo":"zawedcvg/are-large-language-models-attentive-readers","repo_kind":"official","path":"create_prompts_with_fake_para.py","file_url":"https://github.com/zawedcvg/are-large-language-models-attentive-readers/blob/HEAD/create_prompts_with_fake_para.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"187e2e2e5060ca39"}},{"code_sha256_prefix":"2c9129c02bfe4c46","entry":"create_fake_para_openai","repo":"zawedcvg/Are-Large-Language-Models-Attentive-Readers","repo_kind":"official","path":"create_prompts.py","file_url":"https://github.com/zawedcvg/Are-Large-Language-Models-Attentive-Readers/blob/HEAD/create_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2c9129c02bfe4c46"}},{"code_sha256_prefix":"8a8a10f341e59184","entry":"get_required_paragraphs","repo":"zawedcvg/are-large-language-models-attentive-readers","repo_kind":"official","path":"create_prompts_with_fake_para.py","file_url":"https://github.com/zawedcvg/are-large-language-models-attentive-readers/blob/HEAD/create_prompts_with_fake_para.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a8a10f341e59184"}},{"code_sha256_prefix":"7d67f1e6076e1033","entry":"the_mixer","repo":"zawedcvg/Are-Large-Language-Models-Attentive-Readers","repo_kind":"official","path":"create_fake_question_tuples.py","file_url":"https://github.com/zawedcvg/Are-Large-Language-Models-Attentive-Readers/blob/HEAD/create_fake_question_tuples.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d67f1e6076e1033"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}