SPARTQA (SPAtial Reasoning on Textual Question Answering)

SpartQA is a textual question answering benchmark for spatial reasoning on natural language text which contains more realistic spatial phenomena not covered by prior datasets and that is challenging for state-of-the-art language models (LM).

SPARTQA is built on NLVR’s images containing more objects with richer spatial structures. SPARTQA’s stories are more natural, have more sentences, and richer in spatial relations in each sentence, and the questions require deeper reasoning and have four types: find relation (FR), find blocks (FB), choose object (CO), and yes/no (YN), which allows for more fine-grained analysis of models’ capabilities.

https://aclanthology.org/2021.naacl-main.364/