{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learned-token-pruning-for-transformers","title":"Learned Token Pruning for Transformers","arxiv_id":"2107.00910","date":"2021-07-02","proceeding":null,"authors":["Sehoon Kim","Sheng Shen","David Thorsley","Amir Gholami","Woosuk Kwon","Joseph Hassoun","Kurt Keutzer"],"abstract":"Deploying transformer models in practice is challenging due to their inference cost, which scales quadratically with input sequence length. To address this, we present a novel Learned Token Pruning (LTP) method which adaptively removes unimportant tokens as an input sequence passes through transformer layers. In particular, LTP prunes tokens with an attention score below a threshold value which is learned for each layer during training. Our threshold-based method allows the length of the pruned sequence to vary adaptively based on the input sequence, and avoids algorithmically expensive operations such as top-k token selection. We extensively test the performance of LTP on GLUE tasks and show that our method outperforms the prior state-of-the-art token pruning methods by up to ~2.5% higher accuracy with the same amount of FLOPs. In particular, LTP achieves up to 2.1x FLOPs reduction with less than 1% accuracy drop, which results in up to 1.9x and 2.0x throughput improvement on Intel Haswell CPUs and NVIDIA V100 GPUs, respectively. Furthermore, we demonstrate that LTP is more robust than prior methods to variations on input sentence lengths. Our code has been developed in PyTorch and has been open-sourced.","url_abs":"https://arxiv.org/abs/2107.00910v3","url_pdf":"https://arxiv.org/pdf/2107.00910v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learned-token-pruning-for-transformers","repo_url":"https://github.com/kssteven418/ltp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2107.00910","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.00910"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kssteven418/ltp","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":12,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"a727df5fd2f5599a","entry":"check_number_comma","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/convert_slow_tokenizer.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/convert_slow_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a727df5fd2f5599a"}},{"code_sha256_prefix":"d770e1dd1d799a47","entry":"ensure_valid_input","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d770e1dd1d799a47"}},{"code_sha256_prefix":"a4475703ff58ecf9","entry":"gelu_fast","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a4475703ff58ecf9"}},{"code_sha256_prefix":"45bb87451230d5e8","entry":"gelu_new","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"45bb87451230d5e8"}},{"code_sha256_prefix":"7efe3db5c9e2f94d","entry":"generate_identified_filename","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7efe3db5c9e2f94d"}},{"code_sha256_prefix":"f889d53c68539013","entry":"mish","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f889d53c68539013"}},{"code_sha256_prefix":"75992437e8972200","entry":"collect_data","repo":"kssteven418/ltp","repo_kind":"official","path":"make_pibert_paper_plots.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/make_pibert_paper_plots.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"75992437e8972200"}},{"code_sha256_prefix":"5de43db1a351fd89","entry":"convert_slow_tokenizer","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/convert_slow_tokenizer.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/convert_slow_tokenizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5de43db1a351fd89"}},{"code_sha256_prefix":"83995d570ba89e6d","entry":"create_position_ids_from_input_ids","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/models/ltp/modeling_ltp.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/models/ltp/modeling_ltp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"83995d570ba89e6d"}},{"code_sha256_prefix":"ac1bc5cb23b3f2ec","entry":"gelu_fast","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ac1bc5cb23b3f2ec"}},{"code_sha256_prefix":"3449d7d975edfb89","entry":"infer_shapes","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/convert_graph_to_onnx.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/convert_graph_to_onnx.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3449d7d975edfb89"}},{"code_sha256_prefix":"cc8c8c3ebf0c343f","entry":"mish","repo":"kssteven418/ltp","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/kssteven418/ltp/blob/HEAD/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cc8c8c3ebf0c343f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}