{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/corematching-a-co-adaptive-sparse-inference","title":"CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models","arxiv_id":"2505.19235","date":"2025-05-25","proceeding":null,"authors":["Qinsi Wang","Hancheng Ye","Ming-Yu Chung","Yudong Liu","Yueqian Lin","Martin Kuo","Mingyuan Ma","Jianyi Zhang","Yiran Chen"],"abstract":"Vision-Language Models (VLMs) excel across diverse tasks but suffer from high inference costs in time and memory. Token sparsity mitigates inefficiencies in token usage, while neuron sparsity reduces high-dimensional computations, both offering promising solutions to enhance efficiency. Recently, these two sparsity paradigms have evolved largely in parallel, fostering the prevailing assumption that they function independently. However, a fundamental yet underexplored question remains: Do they truly operate in isolation, or is there a deeper underlying interplay that has yet to be uncovered? In this paper, we conduct the first comprehensive investigation into this question. By introducing and analyzing the matching mechanism between Core Neurons and Core Tokens, we found that key neurons and tokens for inference mutually influence and reinforce each other. Building on this insight, we propose CoreMatching, a co-adaptive sparse inference framework, which leverages the synergy between token and neuron sparsity to enhance inference efficiency. Through theoretical analysis and efficiency evaluations, we demonstrate that the proposed method surpasses state-of-the-art baselines on ten image understanding tasks and three hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs reduction and a 10x overall speedup. Code is released at https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main.","url_abs":"https://arxiv.org/abs/2505.19235v1","url_pdf":"https://arxiv.org/pdf/2505.19235v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"corematching-a-co-adaptive-sparse-inference","repo_url":"https://github.com/wangqinsi1/2025-icml-corematching","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.19235","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.19235"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wangqinsi1/2025-icml-corematching","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":10,"ran_honours":1,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":16,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"743ceb9025a7955d","entry":"booleans_processing","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modeling_tf_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modeling_tf_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"743ceb9025a7955d"}},{"code_sha256_prefix":"3c1eb64dde6b86af","entry":"dummy_loss","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modeling_tf_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modeling_tf_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c1eb64dde6b86af"}},{"code_sha256_prefix":"1adcf2d05b2faed1","entry":"get_imports","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/dynamic_module_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1adcf2d05b2faed1"}},{"code_sha256_prefix":"518f6a89865f25d2","entry":"get_relative_import_files","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/dynamic_module_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"518f6a89865f25d2"}},{"code_sha256_prefix":"4af5faa701d0f2c9","entry":"get_relative_imports","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/dynamic_module_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/dynamic_module_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4af5faa701d0f2c9"}},{"code_sha256_prefix":"dd8f7751ef859581","entry":"infer_metric_tags_from_eval_results","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modelcard.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd8f7751ef859581"}},{"code_sha256_prefix":"6a3959a0ffd36577","entry":"keras_serializable","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modeling_tf_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modeling_tf_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a3959a0ffd36577"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"predict.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/predict.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"a4491b45d19b7a39","entry":"parse_keras_history","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modelcard.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a4491b45d19b7a39"}},{"code_sha256_prefix":"a66d2b78670c85e1","entry":"quick_gelu","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modeling_flax_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modeling_flax_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a66d2b78670c85e1"}},{"code_sha256_prefix":"30d50ff2ced6b7a3","entry":"rename_key_and_reshape_tensor","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modeling_flax_pytorch_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modeling_flax_pytorch_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30d50ff2ced6b7a3"}},{"code_sha256_prefix":"55c32993da87759b","entry":"unpad_image","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"55c32993da87759b"}},{"code_sha256_prefix":"b1260d802f5e3997","entry":"convert_pytorch_state_dict_to_flax","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modeling_flax_pytorch_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modeling_flax_pytorch_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1260d802f5e3997"}},{"code_sha256_prefix":"9d98806acb3144b7","entry":"dtype_byte_size","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modeling_flax_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modeling_flax_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d98806acb3144b7"}},{"code_sha256_prefix":"e3ce70ca9258800e","entry":"flax_shard_checkpoint","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modeling_flax_utils.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modeling_flax_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e3ce70ca9258800e"}},{"code_sha256_prefix":"ec7b1528a936234d","entry":"is_hf_dataset","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"transformers/modelcard.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/transformers/modelcard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec7b1528a936234d"}},{"code_sha256_prefix":"522afabbcdf0f218","entry":"load_image","repo":"wangqinsi1/2025-ICML-CoreMatching","repo_kind":"official","path":"inference.py","file_url":"https://github.com/wangqinsi1/2025-ICML-CoreMatching/blob/HEAD/inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"522afabbcdf0f218"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}