{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vision-centric-token-compression-in-large","title":"Vision-centric Token Compression in Large Language Model","arxiv_id":"2502.00791","date":"2025-02-02","proceeding":null,"authors":["Ling Xing","Alex Jinpeng Wang","Rui Yan","Xiangbo Shu","Jinhui Tang"],"abstract":"Real-world applications are stretching context windows to hundreds of thousand of tokens while Large Language Models (LLMs) swell from billions to trillions of parameters. This dual expansion send compute and memory costs skyrocketing, making token compression indispensable. We introduce Vision Centric Token Compression (Vist), a slow-fast compression framework that mirrors human reading: the fast path renders distant tokens into images, letting a frozen, lightweight vision encoder skim the low-salience context; the slow path feeds the proximal window into the LLM for fine-grained reasoning. A Probability-Informed Visual Enhancement (PVE) objective masks high-frequency tokens during training, steering the Resampler to concentrate on semantically rich regions-just as skilled reader gloss over function words. On eleven in-context learning benchmarks, Vist achieves the same accuracy with 2.3 times fewer tokens, cutting FLOPs by 16% and memory by 50%. This method delivers remarkable results, outperforming the strongest text encoder-based compression method CEPE by 7.6% on average over benchmarks like TriviaQA, NQ, PopQA, NLUI, and CLIN, setting a new standard for token efficiency in LLMs. The source code will be released.","url_abs":"https://arxiv.org/abs/2502.00791v3","url_pdf":"https://arxiv.org/pdf/2502.00791v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"triviaqa","task_name":"TriviaQA"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.00791","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.00791"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/CSU-JPG/VIST","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"found_in_text":{"samples":9,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"20b949020615f0c3","entry":"EmbedToLatents","repo":"CSU-JPG/VIST","repo_kind":"found_in_text","path":"multimodal_model/model_architecture.py","file_url":"https://github.com/CSU-JPG/VIST/blob/HEAD/multimodal_model/model_architecture.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20b949020615f0c3"}},{"code_sha256_prefix":"3f110f8fcd0e3d4e","entry":"_expand_mask","repo":"CSU-JPG/VIST","repo_kind":"found_in_text","path":"multimodal_model/model_architecture.py","file_url":"https://github.com/CSU-JPG/VIST/blob/HEAD/multimodal_model/model_architecture.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f110f8fcd0e3d4e"}},{"code_sha256_prefix":"cd5710e6dadab009","entry":"CausalLMOutputWithPastContext","repo":"CSU-JPG/VIST","repo_kind":"found_in_text","path":"multimodal_model/model_architecture.py","file_url":"https://github.com/CSU-JPG/VIST/blob/HEAD/multimodal_model/model_architecture.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd5710e6dadab009"}},{"code_sha256_prefix":"a6d82856d218be13","entry":"GroupAllGather","repo":"CSU-JPG/VIST","repo_kind":"found_in_text","path":"multimodal_model/model_architecture.py","file_url":"https://github.com/CSU-JPG/VIST/blob/HEAD/multimodal_model/model_architecture.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a6d82856d218be13"}},{"code_sha256_prefix":"e9b20c73c25702be","entry":"LlamaForCausalContextLM","repo":"CSU-JPG/VIST","repo_kind":"found_in_text","path":"multimodal_model/model_architecture.py","file_url":"https://github.com/CSU-JPG/VIST/blob/HEAD/multimodal_model/model_architecture.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9b20c73c25702be"}},{"code_sha256_prefix":"b1eb9481dae931d0","entry":"PerceiverAttention","repo":"CSU-JPG/VIST","repo_kind":"found_in_text","path":"multimodal_model/model_architecture.py","file_url":"https://github.com/CSU-JPG/VIST/blob/HEAD/multimodal_model/model_architecture.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1eb9481dae931d0"}},{"code_sha256_prefix":"534c99afd5f49f18","entry":"PerceiverResampler","repo":"CSU-JPG/VIST","repo_kind":"found_in_text","path":"multimodal_model/model_architecture.py","file_url":"https://github.com/CSU-JPG/VIST/blob/HEAD/multimodal_model/model_architecture.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"534c99afd5f49f18"}},{"code_sha256_prefix":"d28e72b052b5d438","entry":"gather_tensors_all_nodes","repo":"CSU-JPG/VIST","repo_kind":"found_in_text","path":"multimodal_model/model_architecture.py","file_url":"https://github.com/CSU-JPG/VIST/blob/HEAD/multimodal_model/model_architecture.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d28e72b052b5d438"}},{"code_sha256_prefix":"f1f89b7214c45bbf","entry":"group_all_gather","repo":"CSU-JPG/VIST","repo_kind":"found_in_text","path":"multimodal_model/model_architecture.py","file_url":"https://github.com/CSU-JPG/VIST/blob/HEAD/multimodal_model/model_architecture.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1f89b7214c45bbf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}