{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tabert-pretraining-for-joint-understanding-of","title":"TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data","arxiv_id":"2005.08314","date":"2020-05-17","proceeding":"ACL 2020 6","authors":["Pengcheng Yin","Graham Neubig","Wen-tau Yih","Sebastian Riedel"],"abstract":"Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider. Implementation of the model will be available at http://fburl.com/TaBERT .","url_abs":"https://arxiv.org/abs/2005.08314v1","url_pdf":"https://arxiv.org/pdf/2005.08314v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tabert-pretraining-for-joint-understanding-of","repo_url":"https://github.com/facebookresearch/tabert","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"form","task_name":"Form"},{"task_slug":"semantic-parsing","task_name":"Semantic Parsing"},{"task_slug":"text-to-sql","task_name":"Text to SQL"},{"task_slug":"text-to-sql","task_name":"Text-To-SQL"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gradient-clipping","method_name":"Gradient Clipping"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"tabert","method_name":"TaBERT"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[{"slug":"tabert","name":"TaBERT","full_name":"TaBERT"}],"results":[{"leaderboard":"/sota/semantic-parsing-on-wikitablequestions","task":"Semantic Parsing","dataset":"WikiTableQuestions","model":"MAPO + TABERTLarge (K = 3)","rank_in_archive_order":18,"of":22,"metrics":{"Accuracy (Dev)":"52.2","Accuracy (Test)":"51.8"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-sql-on-spider","task":"Text-To-SQL","dataset":"spider","model":"MAPO + TABERTLarge (K = 3)","rank_in_archive_order":20,"of":20,"metrics":{"Exact Match Accuracy (Dev)":"64.5"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2005.08314","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}