{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-augmentation-for-bert-fine-tuning-in","title":"Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering","arxiv_id":"1904.06652","date":"2019-04-14","proceeding":null,"authors":["Wei Yang","Yuqing Xie","Luchen Tan","Kun Xiong","Ming Li","Jimmy Lin"],"abstract":"Recently, a simple combination of passage retrieval using off-the-shelf IR\ntechniques and a BERT reader was found to be very effective for question\nanswering directly on Wikipedia, yielding a large improvement over the previous\nstate of the art on a standard benchmark dataset. In this paper, we present a\ndata augmentation technique using distant supervision that exploits positive as\nwell as negative examples. We apply a stage-wise approach to fine tuning BERT\non multiple datasets, starting with data that is \"furthest\" from the test data\nand ending with the \"closest\". Experimental results show large gains in\neffectiveness over previous approaches on English QA datasets, and we establish\nnew baselines on two recent Chinese QA datasets.","url_abs":"http://arxiv.org/abs/1904.06652v1","url_pdf":"http://arxiv.org/pdf/1904.06652v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"open-domain-question-answering","task_name":"Open-Domain Question Answering"},{"task_slug":"passage-retrieval","task_name":"Passage Retrieval"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-domain-question-answering-on-squad1-1","task":"Open-Domain Question Answering","dataset":"SQuAD1.1 dev","model":"BERTserini","rank_in_archive_order":3,"of":4,"metrics":{"EM":"50.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1904.06652","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}