{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/answering-questions-by-meta-reasoning-over","title":"Answering Questions by Meta-Reasoning over Multiple Chains of Thought","arxiv_id":"2304.13007","date":"2023-04-25","proceeding":null,"authors":["Ori Yoran","Tomer Wolfson","Ben Bogin","Uri Katz","Daniel Deutch","Jonathan Berant"],"abstract":"Modern systems for multi-hop question answering (QA) typically break questions into a sequence of reasoning steps, termed chain-of-thought (CoT), before arriving at a final answer. Often, multiple chains are sampled and aggregated through a voting mechanism over the final answers, but the intermediate steps themselves are discarded. While such approaches improve performance, they do not consider the relations between intermediate steps across chains and do not provide a unified explanation for the predicted answer. We introduce Multi-Chain Reasoning (MCR), an approach which prompts large language models to meta-reason over multiple chains of thought, rather than aggregating their answers. MCR examines different reasoning chains, mixes information between them and selects the most relevant facts in generating an explanation and predicting the answer. MCR outperforms strong baselines on 7 multi-hop QA datasets. Moreover, our analysis reveals that MCR explanations exhibit high quality, enabling humans to verify its answers.","url_abs":"https://arxiv.org/abs/2304.13007v4","url_pdf":"https://arxiv.org/pdf/2304.13007v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"answering-questions-by-meta-reasoning-over","repo_url":"https://github.com/oriyor/reasoning-on-cots","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multi-hop-question-answering","task_name":"Multi-hop Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-bamboogle","task":"Question Answering","dataset":"Bamboogle","model":"MCR (code-davinci-002) + Google Search","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"66.5"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2304.13007","atlas_url":"https://app.syntology.ai/?focus=2304.13007","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.13007"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/oriyor/reasoning-on-cots","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"official":{"samples":10,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"86dcd04f5c5623f3","entry":"compute_exact_match","repo":"oriyor/reasoning-on-cots","repo_kind":"official","path":"src/pred_evaluators/evaluation.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/pred_evaluators/evaluation.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"86dcd04f5c5623f3"}},{"code_sha256_prefix":"c30d6b506e3a5d6e","entry":"normalize_text","repo":"oriyor/reasoning-on-cots","repo_kind":"official","path":"src/pred_evaluators/evaluation.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/pred_evaluators/evaluation.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c30d6b506e3a5d6e"}},{"code_sha256_prefix":"63cfc8eb0c95d0dc","entry":"call_gpt","repo":"oriyor/reasoning-on-cots","repo_kind":"official","path":"src/opeanai/utils.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/opeanai/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63cfc8eb0c95d0dc"}},{"code_sha256_prefix":"df23eac515d0b974","entry":"chunk_list_to_sublists","repo":"oriyor/reasoning-on-cots","repo_kind":"official","path":"src/common/dataset_utils.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/common/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df23eac515d0b974"}},{"code_sha256_prefix":"9118271336e0821a","entry":"compute_f1","repo":"oriyor/reasoning-on-cots","repo_kind":"official","path":"src/pred_evaluators/evaluation.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/pred_evaluators/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9118271336e0821a"}},{"code_sha256_prefix":"558f49bd6261449f","entry":"file_as_string","repo":"oriyor/reasoning-on-cots","repo_kind":"official","path":"src/common/dataset_utils.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/common/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"558f49bd6261449f"}},{"code_sha256_prefix":"787934785bc5bf80","entry":"format_decompsition_break","repo":"oriyor/reasoning-on-cots","repo_kind":"official","path":"src/dataclasses.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/dataclasses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"787934785bc5bf80"}},{"code_sha256_prefix":"3b07f989f0ffe622","entry":"greenify","repo":"oriyor/reasoning-on-cots","repo_kind":"official","path":"src/opeanai/utils.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/opeanai/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b07f989f0ffe622"}},{"code_sha256_prefix":"10ff4bd5646b3677","entry":"read_csv_to_dict","repo":"oriyor/reasoning-on-cots","repo_kind":"official","path":"src/common/dataset_utils.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/common/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10ff4bd5646b3677"}},{"code_sha256_prefix":"5cd3899a28826ba7","entry":"yellowfy","repo":"oriyor/reasoning-on-cots","repo_kind":"official","path":"src/opeanai/utils.py","file_url":"https://github.com/oriyor/reasoning-on-cots/blob/HEAD/src/opeanai/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cd3899a28826ba7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}