{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neuro-inspired-fragmentation-and-recall-to","title":"Neuro-Inspired Fragmentation and Recall to Overcome Catastrophic Forgetting in Curiosity","arxiv_id":"2310.17537","date":"2023-10-26","proceeding":null,"authors":["Jaedong Hwang","Zhang-Wei Hong","Eric Chen","Akhilan Boopathy","Pulkit Agrawal","Ila Fiete"],"abstract":"Deep reinforcement learning methods exhibit impressive performance on a range of tasks but still struggle on hard exploration tasks in large environments with sparse rewards. To address this, intrinsic rewards can be generated using forward model prediction errors that decrease as the environment becomes known, and incentivize an agent to explore novel states. While prediction-based intrinsic rewards can help agents solve hard exploration tasks, they can suffer from catastrophic forgetting and actually increase at visited states. We first examine the conditions and causes of catastrophic forgetting in grid world environments. We then propose a new method FARCuriosity, inspired by how humans and animals learn. The method depends on fragmentation and recall: an agent fragments an environment based on surprisal, and uses different local curiosity modules (prediction-based intrinsic reward functions) for each fragment so that modules are not trained on the entire environment. At each fragmentation event, the agent stores the current module in long-term memory (LTM) and either initializes a new module or recalls a previously stored module based on its match with the current state. With fragmentation and recall, FARCuriosity achieves less forgetting and better overall performance in games with varied and heterogeneous environments in the Atari benchmark suite of tasks. Thus, this work highlights the problem of catastrophic forgetting in prediction-based curiosity methods and proposes a solution.","url_abs":"https://arxiv.org/abs/2310.17537v1","url_pdf":"https://arxiv.org/pdf/2310.17537v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neuro-inspired-fragmentation-and-recall-to","repo_url":"https://github.com/fietelab/farcuriosity","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"prediction","task_name":"Prediction"}],"methods":[{"method_slug":"fragmentation","method_name":"Fragmentation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.17537","atlas_url":"https://app.syntology.ai/?focus=2310.17537","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.17537"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fietelab/farcuriosity","reach":null}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2e98e983ba2bc6cc","entry":"Flatten","repo":"fietelab/farcuriosity","repo_kind":"official","path":"rlpyt/models/curiosity/fragmentation_wrapper.py","file_url":"https://github.com/fietelab/farcuriosity/blob/HEAD/rlpyt/models/curiosity/fragmentation_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e98e983ba2bc6cc"}},{"code_sha256_prefix":"3e152d9242f7ae00","entry":"RND","repo":"fietelab/farcuriosity","repo_kind":"official","path":"rlpyt/models/curiosity/fragmentation_wrapper.py","file_url":"https://github.com/fietelab/farcuriosity/blob/HEAD/rlpyt/models/curiosity/fragmentation_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e152d9242f7ae00"}},{"code_sha256_prefix":"72acf65238123601","entry":"RewardForwardFilter","repo":"fietelab/farcuriosity","repo_kind":"official","path":"rlpyt/models/curiosity/fragmentation_wrapper.py","file_url":"https://github.com/fietelab/farcuriosity/blob/HEAD/rlpyt/models/curiosity/fragmentation_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72acf65238123601"}},{"code_sha256_prefix":"c9b232c9c0357ca3","entry":"RunningMeanStd","repo":"fietelab/farcuriosity","repo_kind":"official","path":"rlpyt/models/curiosity/fragmentation_wrapper.py","file_url":"https://github.com/fietelab/farcuriosity/blob/HEAD/rlpyt/models/curiosity/fragmentation_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9b232c9c0357ca3"}},{"code_sha256_prefix":"827c5e17b2cf1164","entry":"RunningStats","repo":"fietelab/farcuriosity","repo_kind":"official","path":"rlpyt/models/curiosity/fragmentation_wrapper.py","file_url":"https://github.com/fietelab/farcuriosity/blob/HEAD/rlpyt/models/curiosity/fragmentation_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"827c5e17b2cf1164"}},{"code_sha256_prefix":"a9909ab5f1d39168","entry":"FragmentationWrapper","repo":"fietelab/farcuriosity","repo_kind":"official","path":"rlpyt/models/curiosity/fragmentation_wrapper.py","file_url":"https://github.com/fietelab/farcuriosity/blob/HEAD/rlpyt/models/curiosity/fragmentation_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9909ab5f1d39168"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}