{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/xrag-extreme-context-compression-for","title":"xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token","arxiv_id":"2405.13792","date":"2024-05-22","proceeding":null,"authors":["Xin Cheng","Xun Wang","Xingxing Zhang","Tao Ge","Si-Qing Chen","Furu Wei","Huishuai Zhang","Dongyan Zhao"],"abstract":"This paper introduces xRAG, an innovative context compression method tailored for retrieval-augmented generation. xRAG reinterprets document embeddings in dense retrieval--traditionally used solely for retrieval--as features from the retrieval modality. By employing a modality fusion methodology, xRAG seamlessly integrates these embeddings into the language model representation space, effectively eliminating the need for their textual counterparts and achieving an extreme compression rate. In xRAG, the only trainable component is the modality bridge, while both the retriever and the language model remain frozen. This design choice allows for the reuse of offline-constructed document embeddings and preserves the plug-and-play nature of retrieval augmentation. Experimental results demonstrate that xRAG achieves an average improvement of over 10% across six knowledge-intensive tasks, adaptable to various language model backbones, ranging from a dense 7B model to an 8x7B Mixture of Experts configuration. xRAG not only significantly outperforms previous context compression methods but also matches the performance of uncompressed models on several datasets, while reducing overall FLOPs by a factor of 3.53. Our work pioneers new directions in retrieval-augmented generation from the perspective of multimodality fusion, and we hope it lays the foundation for future efficient and scalable retrieval-augmented systems","url_abs":"https://arxiv.org/abs/2405.13792v2","url_pdf":"https://arxiv.org/pdf/2405.13792v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"xrag-extreme-context-compression-for","repo_url":"https://github.com/Hannibal046/xRAG","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.13792","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.13792"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Hannibal046/xRAG","reach":null}],"summary":{"ran":1,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"049569a103f41bb3","entry":"Projector","repo":"Hannibal046/xRAG","repo_kind":"official","path":"src/model/xMistral/modeling_xmistral.py","file_url":"https://github.com/Hannibal046/xRAG/blob/HEAD/src/model/xMistral/modeling_xmistral.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"049569a103f41bb3"}},{"code_sha256_prefix":"f32346bc3e2dcdc9","entry":"create_prompt_with_mistral_chat_format","repo":"Hannibal046/xRAG","repo_kind":"official","path":"src/eval/run_eval.py","file_url":"https://github.com/Hannibal046/xRAG/blob/HEAD/src/eval/run_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f32346bc3e2dcdc9"}},{"code_sha256_prefix":"b07caba17b6e0392","entry":"get_start_prompt","repo":"Hannibal046/xRAG","repo_kind":"official","path":"src/eval/run_eval.py","file_url":"https://github.com/Hannibal046/xRAG/blob/HEAD/src/eval/run_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b07caba17b6e0392"}},{"code_sha256_prefix":"159faffa7ba9a59e","entry":"XMistralForCausalLM","repo":"Hannibal046/xRAG","repo_kind":"official","path":"src/model/xMistral/modeling_xmistral.py","file_url":"https://github.com/Hannibal046/xRAG/blob/HEAD/src/model/xMistral/modeling_xmistral.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"159faffa7ba9a59e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}