{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generative-modeling-with-explicit-memory","title":"Generative Modeling with Explicit Memory","arxiv_id":"2412.08781","date":"2024-12-11","proceeding":null,"authors":["Yi Tang","Peng Sun","Zhenglin Cheng","Tao Lin"],"abstract":"Recent studies indicate that the denoising process in deep generative diffusion models implicitly learns and memorizes semantic information from the data distribution. These findings suggest that capturing more complex data distributions requires larger neural networks, leading to a substantial increase in computational demands, which in turn become the primary bottleneck in both training and inference of diffusion models. To this end, we introduce \\textbf{G}enerative \\textbf{M}odeling with \\textbf{E}xplicit \\textbf{M}emory (GMem), leveraging an external memory bank in both training and sampling phases of diffusion models. This approach preserves semantic information from data distributions, reducing reliance on neural network capacity for learning and generalizing across diverse datasets. The results are significant: our GMem enhances both training, sampling efficiency, and generation quality. For instance, on ImageNet at $256 \\times 256$ resolution, GMem accelerates SiT training by over $46.7\\times$, achieving the performance of a SiT model trained for $7M$ steps in fewer than $150K$ steps. Compared to the most efficient existing method, REPA, GMem still offers a $16\\times$ speedup, attaining an FID score of 5.75 within $250K$ steps, whereas REPA requires over $4M$ steps. Additionally, our method achieves state-of-the-art generation quality, with an FID score of {3.56} without classifier-free guidance on ImageNet $256\\times256$. Our code is available at \\url{https://github.com/LINs-lab/GMem}.","url_abs":"https://arxiv.org/abs/2412.08781v1","url_pdf":"https://arxiv.org/pdf/2412.08781v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"generative-modeling-with-explicit-memory","repo_url":"https://github.com/lins-lab/gmem","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-generation-on-cifar-10","task":"Image Generation","dataset":"CIFAR-10","model":"GMem","rank_in_archive_order":1,"of":78,"metrics":{"FID":"1.22"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-256x256","task":"Image Generation","dataset":"ImageNet 256x256","model":"GMem  (with the guidance interval)","rank_in_archive_order":11,"of":94,"metrics":{"FID":"1.32"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-256x256","task":"Image Generation","dataset":"ImageNet 256x256","model":"GMem (w/o guidance)","rank_in_archive_order":22,"of":94,"metrics":{"FID":"1.53"},"uses_additional_data":false},{"leaderboard":"/sota/image-generation-on-imagenet-512x512","task":"Image Generation","dataset":"ImageNet 512x512","model":"GMem","rank_in_archive_order":15,"of":52,"metrics":{"FID":"1.71"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.08781","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.08781"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lins-lab/gmem","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran_honours":2,"ran_draft_wrong":3,"ran_violates":1,"unverified":7},"by_repo_kind":{"official":{"samples":15,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"d7b6dcfe63bfe59b","entry":"apply_rotary_emb","repo":"lins-lab/gmem","repo_kind":"official","path":"models/rmsnorm.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/rmsnorm.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d7b6dcfe63bfe59b"}},{"code_sha256_prefix":"f50a8d20efb35c68","entry":"broadcat","repo":"lins-lab/gmem","repo_kind":"official","path":"models/pos_embed.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/pos_embed.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f50a8d20efb35c68"}},{"code_sha256_prefix":"b05a447cc3b9e991","entry":"build_mlp","repo":"lins-lab/gmem","repo_kind":"official","path":"models/lightningdit.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/lightningdit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b05a447cc3b9e991"}},{"code_sha256_prefix":"c92c27c924b517e8","entry":"get_2d_sincos_pos_embed","repo":"lins-lab/gmem","repo_kind":"official","path":"models/lightningdit.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/lightningdit.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c92c27c924b517e8"}},{"code_sha256_prefix":"7257fdf155f5ef45","entry":"normalize_tensor","repo":"lins-lab/gmem","repo_kind":"official","path":"models/lpips.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/lpips.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7257fdf155f5ef45"}},{"code_sha256_prefix":"04a1fa63d6d4b8e4","entry":"precompute_freqs_cis","repo":"lins-lab/gmem","repo_kind":"official","path":"models/rmsnorm.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/rmsnorm.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04a1fa63d6d4b8e4"}},{"code_sha256_prefix":"5a639d78ada17fee","entry":"reshape_for_broadcast","repo":"lins-lab/gmem","repo_kind":"official","path":"models/rmsnorm.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/rmsnorm.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a639d78ada17fee"}},{"code_sha256_prefix":"58823d9435a8751b","entry":"rotate_half","repo":"lins-lab/gmem","repo_kind":"official","path":"models/pos_embed.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/pos_embed.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58823d9435a8751b"}},{"code_sha256_prefix":"1fb53f7800136b03","entry":"get_ckpt_path","repo":"lins-lab/gmem","repo_kind":"official","path":"models/lpips.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/lpips.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fb53f7800136b03"}},{"code_sha256_prefix":"77589367db56d8df","entry":"get_dino_v2_representation","repo":"lins-lab/gmem","repo_kind":"official","path":"models/dinov2.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/dinov2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77589367db56d8df"}},{"code_sha256_prefix":"020b55cbd4b73888","entry":"image_from_minus_one_to_one","repo":"lins-lab/gmem","repo_kind":"official","path":"models/dinov2.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/dinov2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"020b55cbd4b73888"}},{"code_sha256_prefix":"c8d9d38b144e401f","entry":"md5_hash","repo":"lins-lab/gmem","repo_kind":"official","path":"models/lpips.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/lpips.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8d9d38b144e401f"}},{"code_sha256_prefix":"5ea6b9bd142d4069","entry":"modulate","repo":"lins-lab/gmem","repo_kind":"official","path":"models/lightningdit.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/lightningdit.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ea6b9bd142d4069"}},{"code_sha256_prefix":"857c29aa550ee7a7","entry":"parse_args","repo":"lins-lab/gmem","repo_kind":"official","path":"datasets/construct_memory_bank.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/datasets/construct_memory_bank.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"857c29aa550ee7a7"}},{"code_sha256_prefix":"eaac1bca2aa9d602","entry":"preprocess_raw_image","repo":"lins-lab/gmem","repo_kind":"official","path":"models/dinov2.py","file_url":"https://github.com/lins-lab/gmem/blob/HEAD/models/dinov2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eaac1bca2aa9d602"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}