{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pre-train-or-annotate-domain-adaptation-with","title":"Pre-train or Annotate? Domain Adaptation with a Constrained Budget","arxiv_id":"2109.04711","date":"2021-09-10","proceeding":"EMNLP 2021 11","authors":["Fan Bai","Alan Ritter","Wei Xu"],"abstract":"Recent work has demonstrated that pre-training in-domain language models can boost performance when adapting to a new domain. However, the costs associated with pre-training raise an important question: given a fixed budget, what steps should an NLP practitioner take to maximize performance? In this paper, we view domain adaptation with a constrained budget as a consumer choice problem, where the goal is to select an optimal combination of data annotation and pre-training. We measure annotation costs of three procedural text datasets, along with the pre-training costs of several in-domain language models. The utility of different combinations of pre-training and data annotation are evaluated under varying budget constraints to assess which combination strategy works best. We find that for small budgets, spending all funds on annotation leads to the best performance; once the budget becomes large enough, however, a combination of data annotation and in-domain pre-training yields better performance. Our experiments suggest task-specific data annotation should be part of an economical strategy when adapting an NLP model to a new domain.","url_abs":"https://arxiv.org/abs/2109.04711v3","url_pdf":"https://arxiv.org/pdf/2109.04711v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pre-train-or-annotate-domain-adaptation-with","repo_url":"https://github.com/bflashcp3f/procbert","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.04711","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.04711"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/allenai/kb","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/saffsd/langid.py","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/bflashcp3f/ProcBERT","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/nlplab/brat","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/allenai/tpu_","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"found_in_text":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e4736e9992d1de49","entry":"is_chunk_end","repo":"bflashcp3f/ProcBERT","repo_kind":"official","path":"code/utils/conlleval.py","file_url":"https://github.com/bflashcp3f/ProcBERT/blob/HEAD/code/utils/conlleval.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4736e9992d1de49"}},{"code_sha256_prefix":"405fd06f23bf2a87","entry":"is_chunk_start","repo":"bflashcp3f/ProcBERT","repo_kind":"official","path":"code/utils/conlleval.py","file_url":"https://github.com/bflashcp3f/ProcBERT/blob/HEAD/code/utils/conlleval.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"405fd06f23bf2a87"}},{"code_sha256_prefix":"f9a22a5ce23105fd","entry":"split_tag","repo":"bflashcp3f/ProcBERT","repo_kind":"official","path":"code/utils/conlleval.py","file_url":"https://github.com/bflashcp3f/ProcBERT/blob/HEAD/code/utils/conlleval.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9a22a5ce23105fd"}},{"code_sha256_prefix":"9cb44938e773d88b","entry":"score","repo":"allenai/kb","repo_kind":"found_in_text","path":"bin/tacred_scorer.py","file_url":"https://github.com/allenai/kb/blob/HEAD/bin/tacred_scorer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9cb44938e773d88b"}},{"code_sha256_prefix":"9ac58585dd89748b","entry":"unpack_wsd_training_instance","repo":"allenai/kb","repo_kind":"found_in_text","path":"kb/wordnet.py","file_url":"https://github.com/allenai/kb/blob/HEAD/kb/wordnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ac58585dd89748b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}