{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evopress-towards-optimal-dynamic-model","title":"EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary Search","arxiv_id":"2410.14649","date":"2024-10-18","proceeding":null,"authors":["Oliver Sieberling","Denis Kuznedelev","Eldar Kurtic","Dan Alistarh"],"abstract":"The high computational costs of large language models (LLMs) have led to a flurry of research on LLM compression, via methods such as quantization, sparsification, or structured pruning. A new frontier in this area is given by \\emph{dynamic, non-uniform} compression methods, which adjust the compression levels (e.g., sparsity) per-block or even per-layer in order to minimize accuracy loss, while guaranteeing a global compression threshold. Yet, current methods rely on heuristics for identifying the \"importance\" of a given layer towards the loss, based on assumptions such as \\emph{error monotonicity}, i.e. that the end-to-end model compression error is proportional to the sum of layer-wise errors. In this paper, we revisit this area, and propose a new and general approach for dynamic compression that is provably optimal in a given input range. We begin from the motivating observation that, in general, \\emph{error monotonicity does not hold for LLMs}: compressed models with lower sum of per-layer errors can perform \\emph{worse} than models with higher error sums. To address this, we propose a new general evolutionary framework for dynamic LLM compression called EvoPress, which has provable convergence, and low sample and evaluation complexity. We show that these theoretical guarantees lead to highly competitive practical performance for dynamic compression of Llama, Mistral and Phi models. Via EvoPress, we set new state-of-the-art results across all compression approaches: structural pruning (block/layer dropping), unstructured sparsity, as well as quantization with dynamic bitwidths. Our code is available at https://github.com/IST-DASLab/EvoPress.","url_abs":"https://arxiv.org/abs/2410.14649v1","url_pdf":"https://arxiv.org/pdf/2410.14649v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evopress-towards-optimal-dynamic-model","repo_url":"https://github.com/ist-daslab/evopress","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.14649","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.14649"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/IST-DASLab/EvoPress","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ist-daslab/evopress","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"ran_fixture":1,"ran_draft_wrong":4,"unverified":7},"by_repo_kind":{"official":{"samples":17,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"224d81b67a1c2965","entry":"compute_cosine_similarity","repo":"IST-DASLab/EvoPress","repo_kind":"official","path":"drop_scoring.py","file_url":"https://github.com/IST-DASLab/EvoPress/blob/HEAD/drop_scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"224d81b67a1c2965"}},{"code_sha256_prefix":"a03e24b71865a274","entry":"compute_fitness","repo":"ist-daslab/evopress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/ist-daslab/evopress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a03e24b71865a274"}},{"code_sha256_prefix":"a7fde36611f443e7","entry":"compute_kl_div","repo":"ist-daslab/evopress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/ist-daslab/evopress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7fde36611f443e7"}},{"code_sha256_prefix":"5e9ff500210ac761","entry":"compute_l2_error","repo":"IST-DASLab/EvoPress","repo_kind":"official","path":"drop_scoring.py","file_url":"https://github.com/IST-DASLab/EvoPress/blob/HEAD/drop_scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5e9ff500210ac761"}},{"code_sha256_prefix":"4ba54cabefb848f1","entry":"compute_l2_error_normalized","repo":"IST-DASLab/EvoPress","repo_kind":"official","path":"drop_scoring.py","file_url":"https://github.com/IST-DASLab/EvoPress/blob/HEAD/drop_scoring.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4ba54cabefb848f1"}},{"code_sha256_prefix":"e1d3ed66ce398037","entry":"compute_perplexity","repo":"ist-daslab/evopress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/ist-daslab/evopress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1d3ed66ce398037"}},{"code_sha256_prefix":"e530b636de6951b7","entry":"get_attn_layer_name","repo":"ist-daslab/evopress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/ist-daslab/evopress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e530b636de6951b7"}},{"code_sha256_prefix":"ae2d0c398eb4627a","entry":"get_legal_mask","repo":"IST-DASLab/EvoPress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/IST-DASLab/EvoPress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae2d0c398eb4627a"}},{"code_sha256_prefix":"b0abeec22706332c","entry":"get_mlp_layer_name","repo":"ist-daslab/evopress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/ist-daslab/evopress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0abeec22706332c"}},{"code_sha256_prefix":"51b6841c58c15248","entry":"is_valid_state","repo":"IST-DASLab/EvoPress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/IST-DASLab/EvoPress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"51b6841c58c15248"}},{"code_sha256_prefix":"5288f0e7d0a12f03","entry":"get_layer_drop_config","repo":"IST-DASLab/EvoPress","repo_kind":"official","path":"brute_force_drop.py","file_url":"https://github.com/IST-DASLab/EvoPress/blob/HEAD/brute_force_drop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5288f0e7d0a12f03"}},{"code_sha256_prefix":"0611818d879db651","entry":"load_compressed_weights","repo":"IST-DASLab/EvoPress","repo_kind":"official","path":"eval_ppl.py","file_url":"https://github.com/IST-DASLab/EvoPress/blob/HEAD/eval_ppl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0611818d879db651"}},{"code_sha256_prefix":"5d2893037018209d","entry":"load_compressed_weights","repo":"IST-DASLab/EvoPress","repo_kind":"official","path":"lmeval.py","file_url":"https://github.com/IST-DASLab/EvoPress/blob/HEAD/lmeval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d2893037018209d"}},{"code_sha256_prefix":"2539ab93d1be2ac3","entry":"load_states","repo":"ist-daslab/evopress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/ist-daslab/evopress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2539ab93d1be2ac3"}},{"code_sha256_prefix":"d7a0175db86b90db","entry":"make_dummy_forward","repo":"ist-daslab/evopress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/ist-daslab/evopress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d7a0175db86b90db"}},{"code_sha256_prefix":"3943134b24b7864b","entry":"restore_forward","repo":"ist-daslab/evopress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/ist-daslab/evopress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3943134b24b7864b"}},{"code_sha256_prefix":"e0f2f410f7063ffa","entry":"selection","repo":"ist-daslab/evopress","repo_kind":"official","path":"evo_drop_search.py","file_url":"https://github.com/ist-daslab/evopress/blob/HEAD/evo_drop_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0f2f410f7063ffa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}