{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hmt-hierarchical-memory-transformer-for-long","title":"HMT: Hierarchical Memory Transformer for Long Context Language Processing","arxiv_id":"2405.06067","date":"2024-05-09","proceeding":null,"authors":["Zifan He","Zongyue Qin","Neha Prakriya","Yizhou Sun","Jason Cong"],"abstract":"Transformer-based large language models (LLM) have been widely used in language processing applications. However, most of them restrict the context window that permits the model to attend to every token in the inputs. Previous works in recurrent models can memorize past tokens to enable unlimited context and maintain effectiveness. However, they have \"flat\" memory architectures, which have limitations in selecting and filtering information. Since humans are good at learning and self-adjustment, we speculate that imitating brain memory hierarchy is beneficial for model memorization. We propose the Hierarchical Memory Transformer (HMT), a novel framework that enables and improves models' long-context processing ability by imitating human memorization behavior. Leveraging memory-augmented segment-level recurrence, we organize the memory hierarchy by preserving tokens from early input token segments, passing memory embeddings along the sequence, and recalling relevant information from history. Evaluating general language modeling (Wikitext-103, PG-19) and question-answering tasks (PubMedQA), we show that HMT steadily improves the long-context processing ability of context-constrained and long-context models. With an additional 0.5% - 2% of parameters, HMT can easily plug in and augment future LLMs to handle long context effectively. Our code is open-sourced on Github: https://github.com/OswaldHe/HMT-pytorch.","url_abs":"https://arxiv.org/abs/2405.06067v2","url_pdf":"https://arxiv.org/pdf/2405.06067v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hmt-hierarchical-memory-transformer-for-long","repo_url":"https://github.com/OswaldHe/HMT-pytorch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.06067","atlas_url":"https://app.syntology.ai/?focus=2405.06067","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.06067"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/OswaldHe/HMT-pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"73a56369878e9809","entry":"PubMedQA","repo":"OswaldHe/HMT-pytorch","repo_kind":"official","path":"hmt_src/pubmedqa_ds_preprocess.py","file_url":"https://github.com/OswaldHe/HMT-pytorch/blob/HEAD/hmt_src/pubmedqa_ds_preprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73a56369878e9809"}},{"code_sha256_prefix":"acb3090c9c09b349","entry":"inject_eae","repo":"OswaldHe/HMT-pytorch","repo_kind":"official","path":"modeling_rmt/compression.py","file_url":"https://github.com/OswaldHe/HMT-pytorch/blob/HEAD/modeling_rmt/compression.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"acb3090c9c09b349"}},{"code_sha256_prefix":"ea22b08819eb2608","entry":"LongSFT","repo":"OswaldHe/HMT-pytorch","repo_kind":"official","path":"hmt_src/long_sft_ds_preprocess.py","file_url":"https://github.com/OswaldHe/HMT-pytorch/blob/HEAD/hmt_src/long_sft_ds_preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ea22b08819eb2608"}},{"code_sha256_prefix":"386041385855d2e1","entry":"OpenROAD","repo":"OswaldHe/HMT-pytorch","repo_kind":"official","path":"hmt_src/openroad_qa_preprocess.py","file_url":"https://github.com/OswaldHe/HMT-pytorch/blob/HEAD/hmt_src/openroad_qa_preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"386041385855d2e1"}},{"code_sha256_prefix":"5ae82ad3d97c3278","entry":"OpenROAD_test","repo":"OswaldHe/HMT-pytorch","repo_kind":"official","path":"hmt_src/openroad_qa_preprocess.py","file_url":"https://github.com/OswaldHe/HMT-pytorch/blob/HEAD/hmt_src/openroad_qa_preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ae82ad3d97c3278"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}