{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/constructing-datasets-for-multi-hop-reading","title":"Constructing Datasets for Multi-hop Reading Comprehension Across Documents","arxiv_id":"1710.06481","date":"2017-10-17","proceeding":"TACL 2018 1","authors":["Johannes Welbl","Pontus Stenetorp","Sebastian Riedel"],"abstract":"Most Reading Comprehension methods limit themselves to queries which can be\nanswered using a single sentence, paragraph, or document. Enabling models to\ncombine disjoint pieces of textual evidence would extend the scope of machine\ncomprehension methods, but currently there exist no resources to train and test\nthis capability. We propose a novel task to encourage the development of models\nfor text understanding across multiple documents and to investigate the limits\nof existing methods. In our task, a model learns to seek and combine evidence -\neffectively performing multi-hop (alias multi-step) inference. We devise a\nmethodology to produce datasets for this task, given a collection of\nquery-answer pairs and thematically linked documents. Two datasets from\ndifferent domains are induced, and we identify potential pitfalls and devise\ncircumvention strategies. We evaluate two previously proposed competitive\nmodels and find that one can integrate information across documents. However,\nboth models struggle to select relevant information, as providing documents\nguaranteed to be relevant greatly improves their performance. While the models\noutperform several strong baselines, their best accuracy reaches 42.9% compared\nto human performance at 74.0% - leaving ample room for improvement.","url_abs":"http://arxiv.org/abs/1710.06481v2","url_pdf":"http://arxiv.org/pdf/1710.06481v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"multi-hop-reading-comprehension","task_name":"Multi-Hop Reading Comprehension"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[{"slug":"medhop","name":"MedHop","full_name":"MedHop"},{"slug":"wikihop","name":"WikiHop","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-wikihop","task":"Question Answering","dataset":"WikiHop","model":"BiDAF","rank_in_archive_order":9,"of":9,"metrics":{"Test":"42.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1710.06481","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}