{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stepgame-a-new-benchmark-for-robust-multi-hop-1","title":"StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts","arxiv_id":"2204.08292","date":"2022-04-18","proceeding":"Association for the Advancement of Artificial Intelligence (AAAI) 2022 2","authors":["Zhengxiang Shi","Qiang Zhang","Aldo Lipani"],"abstract":"Inferring spatial relations in natural language is a crucial ability an intelligent system should possess. The bAbI dataset tries to capture tasks relevant to this domain (task 17 and 19). However, these tasks have several limitations. Most importantly, they are limited to fixed expressions, they are limited in the number of reasoning steps required to solve them, and they fail to test the robustness of models to input that contains irrelevant or redundant information. In this paper, we present a new Question-Answering dataset called StepGame for robust multi-hop spatial reasoning in texts. Our experiments demonstrate that state-of-the-art models on the bAbI dataset struggle on the StepGame dataset. Moreover, we propose a Tensor-Product based Memory-Augmented Neural Network (TP-MANN) specialized for spatial reasoning tasks. Experimental results on both datasets show that our model outperforms all the baselines with superior generalization and robustness performance.","url_abs":"https://arxiv.org/abs/2204.08292v1","url_pdf":"https://arxiv.org/pdf/2204.08292v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stepgame-a-new-benchmark-for-robust-multi-hop-1","repo_url":"https://github.com/ZhengxiangShi/StepGame","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"spatial-reasoning","task_name":"Spatial Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-stepgame","task":"Question Answering","dataset":"StepGame","model":"TP-MANN","rank_in_archive_order":1,"of":1,"metrics":{"1-of-100 Accuracy":"52.99"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.08292","atlas_url":"https://app.syntology.ai/?focus=2204.08292","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.08292"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ZhengxiangShi/StepGame","reach":null}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c45368528446001c","entry":"InferenceModule","repo":"ZhengxiangShi/StepGame","repo_kind":"official","path":"Code/TP-MANN/model/tp_mann.py","file_url":"https://github.com/ZhengxiangShi/StepGame/blob/HEAD/Code/TP-MANN/model/tp_mann.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c45368528446001c"}},{"code_sha256_prefix":"8b94a14b1e542957","entry":"LayerNorm","repo":"ZhengxiangShi/StepGame","repo_kind":"official","path":"Code/TP-MANN/model/tp_mann.py","file_url":"https://github.com/ZhengxiangShi/StepGame/blob/HEAD/Code/TP-MANN/model/tp_mann.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b94a14b1e542957"}},{"code_sha256_prefix":"07292c83f8398f4d","entry":"MLP","repo":"ZhengxiangShi/StepGame","repo_kind":"official","path":"Code/TP-MANN/model/tp_mann.py","file_url":"https://github.com/ZhengxiangShi/StepGame/blob/HEAD/Code/TP-MANN/model/tp_mann.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07292c83f8398f4d"}},{"code_sha256_prefix":"3dae9138f98cdefa","entry":"OptionalLayer","repo":"ZhengxiangShi/StepGame","repo_kind":"official","path":"Code/TP-MANN/model/tp_mann.py","file_url":"https://github.com/ZhengxiangShi/StepGame/blob/HEAD/Code/TP-MANN/model/tp_mann.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3dae9138f98cdefa"}},{"code_sha256_prefix":"b61d5d9761232a97","entry":"UpdateModule","repo":"ZhengxiangShi/StepGame","repo_kind":"official","path":"Code/TP-MANN/model/tp_mann.py","file_url":"https://github.com/ZhengxiangShi/StepGame/blob/HEAD/Code/TP-MANN/model/tp_mann.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b61d5d9761232a97"}},{"code_sha256_prefix":"bc83a2dc9f1b6337","entry":"InputModule","repo":"ZhengxiangShi/StepGame","repo_kind":"official","path":"Code/TP-MANN/model/tp_mann.py","file_url":"https://github.com/ZhengxiangShi/StepGame/blob/HEAD/Code/TP-MANN/model/tp_mann.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc83a2dc9f1b6337"}},{"code_sha256_prefix":"ddc0dbfc4e527788","entry":"Tpmann","repo":"ZhengxiangShi/StepGame","repo_kind":"official","path":"Code/TP-MANN/model/tp_mann.py","file_url":"https://github.com/ZhengxiangShi/StepGame/blob/HEAD/Code/TP-MANN/model/tp_mann.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ddc0dbfc4e527788"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}