{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/robut-a-systematic-study-of-table-qa","title":"RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations","arxiv_id":"2306.14321","date":"2023-06-25","proceeding":null,"authors":["Yilun Zhao","Chen Zhao","Linyong Nan","Zhenting Qi","Wenlin Zhang","Xiangru Tang","Boyu Mi","Dragomir Radev"],"abstract":"Despite significant progress having been made in question answering on tabular data (Table QA), it's unclear whether, and to what extent existing Table QA models are robust to task-specific perturbations, e.g., replacing key question entities or shuffling table columns. To systematically study the robustness of Table QA models, we propose a benchmark called RobuT, which builds upon existing Table QA datasets (WTQ, WikiSQL-Weak, and SQA) and includes human-annotated adversarial perturbations in terms of table header, table content, and question. Our results indicate that both state-of-the-art Table QA models and large language models (e.g., GPT-3) with few-shot learning falter in these adversarial sets. We propose to address this problem by using large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models. Our data and code is publicly available at https://github.com/yilunzhao/RobuT.","url_abs":"https://arxiv.org/abs/2306.14321v1","url_pdf":"https://arxiv.org/pdf/2306.14321v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"robut-a-systematic-study-of-table-qa","repo_url":"https://github.com/yilunzhao/robut","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.14321","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.14321"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yilunzhao/robut","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/yilunzhao/RobuT","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1d43704b2478ff62","entry":"compute_prediction_sequence","repo":"yilunzhao/RobuT","repo_kind":"official","path":"main/run_tapas.py","file_url":"https://github.com/yilunzhao/RobuT/blob/HEAD/main/run_tapas.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d43704b2478ff62"}},{"code_sha256_prefix":"bca6148c557e6cfd","entry":"convert_to_float","repo":"yilunzhao/RobuT","repo_kind":"official","path":"utils/eval_utils.py","file_url":"https://github.com/yilunzhao/RobuT/blob/HEAD/utils/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bca6148c557e6cfd"}},{"code_sha256_prefix":"7998bb6f998f1b8a","entry":"execute","repo":"yilunzhao/RobuT","repo_kind":"official","path":"utils/eval_utils.py","file_url":"https://github.com/yilunzhao/RobuT/blob/HEAD/utils/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7998bb6f998f1b8a"}},{"code_sha256_prefix":"f3d19b17e614286d","entry":"to_float32","repo":"yilunzhao/RobuT","repo_kind":"official","path":"utils/eval_utils.py","file_url":"https://github.com/yilunzhao/RobuT/blob/HEAD/utils/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3d19b17e614286d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}