{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-qualitative-comparison-of-coqa-squad-20-and","title":"A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC","arxiv_id":"1809.10735","date":"2018-09-27","proceeding":"NAACL 2019 6","authors":["Mark Yatskar"],"abstract":"We compare three new datasets for question answering: SQuAD 2.0, QuAC, and CoQA, along several of their new features: (1) unanswerable questions, (2) multi-turn interactions, and (3) abstractive answers. We show that the datasets provide complementary coverage of the first two aspects, but weak coverage of the third. Because of the datasets' structural similarity, a single extractive model can be easily adapted to any of the datasets and we show improved baseline results on both SQuAD 2.0 and CoQA. Despite the similarity, models trained on one dataset are ineffective on another dataset, but we find moderate performance improvement through pretraining. To encourage cross-evaluation, we release code for conversion between datasets at https://github.com/my89/co-squac .","url_abs":"https://arxiv.org/abs/1809.10735v2","url_pdf":"https://arxiv.org/pdf/1809.10735v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-qualitative-comparison-of-coqa-squad-20-and","repo_url":"https://github.com/my89/co-squac","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/question-answering-on-coqa","task":"Question Answering","dataset":"CoQA","model":"BiDAF++ (single model)","rank_in_archive_order":3,"of":9,"metrics":{"In-domain":"69.4","Out-of-domain":"63.8","Overall":"67.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1809.10735","atlas_url":"https://app.syntology.ai/?focus=1809.10735","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}