{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-10021","title":"Decoupled Reasoning with Implicit Fact Tokens (DRIFT): A Dual-Model Framework for Efficient Long-Context Inference","arxiv_id":"2602.10021","date":"2026-02-10","proceeding":null,"authors":["Wenxuan Xie","Yujia Wang","Xin Tan","Chaochao Lu","Xia Hu","Xuhong Wang"],"abstract":"The integration of extensive, dynamic knowledge into Large Language Models (LLMs) remains a significant challenge due to the inherent entanglement of factual data and reasoning patterns. Existing solutions, ranging from non-parametric Retrieval-Augmented Generation (RAG) to parametric knowledge editing, are often constrained in practice by finite context windows, retriever noise, or the risk of catastrophic forgetting. In this paper, we propose DRIFT, a novel dual-model architecture designed to explicitly decouple knowledge extraction from the reasoning process. Unlike static prompt compression, DRIFT employs a lightweight knowledge model to dynamically compress document chunks into implicit fact tokens conditioned on the query. These dense representations are projected into the reasoning model's embedding space, replacing raw, redundant text while maintaining inference accuracy. Extensive experiments show that DRIFT significantly improves performance on long-context tasks, outperforming strong baselines among comparably sized models. Our approach provides a scalable and efficient paradigm for extending the effective context window and reasoning capabilities of LLMs. Our code is available at https://github.com/Lancelot-Xie/DRIFT.","url_abs":"https://arxiv.org/abs/2602.10021","url_pdf":"https://arxiv.org/pdf/2602.10021","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.10021","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.10021"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Lancelot-Xie/DRIFT","reach":null}],"summary":{"unverified":15},"by_repo_kind":{"found_in_text":{"samples":15,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"a04afef2512ccc7a","entry":"bucketed_compression_tokens","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/utils/compression.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/utils/compression.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a04afef2512ccc7a"}},{"code_sha256_prefix":"27bd557a2de23b04","entry":"build_projector","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/modeling_drift.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/modeling_drift.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27bd557a2de23b04"}},{"code_sha256_prefix":"cef7009e844c92a4","entry":"calculate_shifted_loss","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/training/losses.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/training/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cef7009e844c92a4"}},{"code_sha256_prefix":"14d7d29ae1462b0a","entry":"create_optimizer","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/training/trainer.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/training/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14d7d29ae1462b0a"}},{"code_sha256_prefix":"ff105ae48c2c98a2","entry":"create_scheduler","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/training/trainer.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/training/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff105ae48c2c98a2"}},{"code_sha256_prefix":"a39128889ad3c700","entry":"cuda_index","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/inference/eval_multi.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/inference/eval_multi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a39128889ad3c700"}},{"code_sha256_prefix":"57e872e92dd63a7c","entry":"get_kl_loss","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/training/losses.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/training/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57e872e92dd63a7c"}},{"code_sha256_prefix":"dd6165b5553efda8","entry":"get_kl_weight","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/training/trainer.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/training/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd6165b5553efda8"}},{"code_sha256_prefix":"0903447d26ca38ef","entry":"normalize_stage","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/utils/stages.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/utils/stages.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0903447d26ca38ef"}},{"code_sha256_prefix":"0e717fa81bd2de66","entry":"parse_json","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"data_generation/generate_qa.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/data_generation/generate_qa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e717fa81bd2de66"}},{"code_sha256_prefix":"36940676e5b7d827","entry":"read_document","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/inference/infer.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/inference/infer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36940676e5b7d827"}},{"code_sha256_prefix":"c35fa39d6741b7b7","entry":"read_jsonl","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/inference/eval_multi.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/inference/eval_multi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c35fa39d6741b7b7"}},{"code_sha256_prefix":"ecfca9ecf998241d","entry":"resolve_drift_checkpoint_paths","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/loading.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/loading.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ecfca9ecf998241d"}},{"code_sha256_prefix":"1ecf3880ae59a9ed","entry":"split_context","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"data_generation/generate_qa.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/data_generation/generate_qa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ecf3880ae59a9ed"}},{"code_sha256_prefix":"d38665d0581e3b19","entry":"to_legacy_phase","repo":"Lancelot-Xie/DRIFT","repo_kind":"found_in_text","path":"src/drift/utils/stages.py","file_url":"https://github.com/Lancelot-Xie/DRIFT/blob/HEAD/src/drift/utils/stages.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d38665d0581e3b19"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}