{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-right-tool-for-the-job-matching-model-and","title":"The Right Tool for the Job: Matching Model and Instance Complexities","arxiv_id":"2004.07453","date":"2020-04-16","proceeding":"ACL 2020 6","authors":["Roy Schwartz","Gabriel Stanovsky","Swabha Swayamdipta","Jesse Dodge","Noah A. Smith"],"abstract":"As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. To better respect a given inference budget, we propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) \"exit\" from neural network calculations for simple instances, and late (and accurate) exit for hard instances. To achieve this, we add classifiers to different layers of BERT and use their calibrated confidence scores to make early exit decisions. We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks. Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing models which are up to five times faster than the state of the art, while preserving their accuracy. Our method also requires almost no additional training resources (in either time or parameters) compared to the baseline BERT model. Finally, our method alleviates the need for costly retraining of multiple models at different levels of efficiency; we allow users to control the inference speed/accuracy tradeoff using a single trained model, by setting a single variable at inference time. We publicly release our code.","url_abs":"https://arxiv.org/abs/2004.07453v2","url_pdf":"https://arxiv.org/pdf/2004.07453v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-right-tool-for-the-job-matching-model-and","repo_url":"https://github.com/allenai/sledgehammer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2004.07453","atlas_url":"https://app.syntology.ai/?focus=2004.07453","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.07453"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/sledgehammer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"allenai/sledgehammer","repo_kind":"official","path":"allennlp_overrides/pytorch_pretrained_bert/modeling.py","file_url":"https://github.com/allenai/sledgehammer/blob/HEAD/allennlp_overrides/pytorch_pretrained_bert/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"cf9ffa02a42184af","entry":"whitespace_tokenize","repo":"allenai/sledgehammer","repo_kind":"official","path":"allennlp_overrides/pytorch_pretrained_bert/tokenization.py","file_url":"https://github.com/allenai/sledgehammer/blob/HEAD/allennlp_overrides/pytorch_pretrained_bert/tokenization.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf9ffa02a42184af"}},{"code_sha256_prefix":"fdc64f4c72036ae4","entry":"gelu","repo":"allenai/sledgehammer","repo_kind":"official","path":"allennlp_overrides/pytorch_pretrained_bert/modeling.py","file_url":"https://github.com/allenai/sledgehammer/blob/HEAD/allennlp_overrides/pytorch_pretrained_bert/modeling.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fdc64f4c72036ae4"}},{"code_sha256_prefix":"5eff22fa0a651276","entry":"url_to_filename","repo":"allenai/sledgehammer","repo_kind":"official","path":"allennlp_overrides/pytorch_pretrained_bert/file_utils.py","file_url":"https://github.com/allenai/sledgehammer/blob/HEAD/allennlp_overrides/pytorch_pretrained_bert/file_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5eff22fa0a651276"}},{"code_sha256_prefix":"cf9c448c96f48428","entry":"cached_path","repo":"allenai/sledgehammer","repo_kind":"official","path":"allennlp_overrides/pytorch_pretrained_bert/file_utils.py","file_url":"https://github.com/allenai/sledgehammer/blob/HEAD/allennlp_overrides/pytorch_pretrained_bert/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf9c448c96f48428"}},{"code_sha256_prefix":"a6aab787fcf8b10d","entry":"filename_to_url","repo":"allenai/sledgehammer","repo_kind":"official","path":"allennlp_overrides/pytorch_pretrained_bert/file_utils.py","file_url":"https://github.com/allenai/sledgehammer/blob/HEAD/allennlp_overrides/pytorch_pretrained_bert/file_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a6aab787fcf8b10d"}},{"code_sha256_prefix":"b1a7180868dd8d6a","entry":"load_tf_weights_in_bert","repo":"allenai/sledgehammer","repo_kind":"official","path":"allennlp_overrides/pytorch_pretrained_bert/modeling.py","file_url":"https://github.com/allenai/sledgehammer/blob/HEAD/allennlp_overrides/pytorch_pretrained_bert/modeling.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1a7180868dd8d6a"}},{"code_sha256_prefix":"a4e183f24a5ad1a7","entry":"load_vocab","repo":"allenai/sledgehammer","repo_kind":"official","path":"allennlp_overrides/pytorch_pretrained_bert/tokenization.py","file_url":"https://github.com/allenai/sledgehammer/blob/HEAD/allennlp_overrides/pytorch_pretrained_bert/tokenization.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a4e183f24a5ad1a7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}