{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/intrinsic-motivation-via-surprise-memory","title":"Beyond Surprise: Improving Exploration Through Surprise Novelty","arxiv_id":"2308.04836","date":"2023-08-09","proceeding":null,"authors":["Hung Le","Kien Do","Dung Nguyen","Svetha Venkatesh"],"abstract":"We present a new computing model for intrinsic rewards in reinforcement learning that addresses the limitations of existing surprise-driven explorations. The reward is the novelty of the surprise rather than the surprise norm. We estimate the surprise novelty as retrieval errors of a memory network wherein the memory stores and reconstructs surprises. Our surprise memory (SM) augments the capability of surprise-based intrinsic motivators, maintaining the agent's interest in exciting exploration while reducing unwanted attraction to unpredictable or noisy observations. Our experiments demonstrate that the SM combined with various surprise predictors exhibits efficient exploring behaviors and significantly boosts the final performance in sparse reward environments, including Noisy-TV, navigation and challenging Atari games.","url_abs":"https://arxiv.org/abs/2308.04836v2","url_pdf":"https://arxiv.org/pdf/2308.04836v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"intrinsic-motivation-via-surprise-memory","repo_url":"https://github.com/thaihungle/sm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"memory-network","method_name":"Memory Network"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2308.04836","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.04836"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thaihungle/sm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"05b20074012b6168","entry":"batched_index_select","repo":"thaihungle/sm","repo_kind":"official","path":"model.py","file_url":"https://github.com/thaihungle/sm/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05b20074012b6168"}},{"code_sha256_prefix":"9de355e93051e4ad","entry":"init","repo":"thaihungle/sm","repo_kind":"official","path":"dist.py","file_url":"https://github.com/thaihungle/sm/blob/HEAD/dist.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9de355e93051e4ad"}},{"code_sha256_prefix":"14c30cad6d60b1b4","entry":"make_train_data","repo":"thaihungle/sm","repo_kind":"official","path":"utils.py","file_url":"https://github.com/thaihungle/sm/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14c30cad6d60b1b4"}},{"code_sha256_prefix":"011537d0bea7fe09","entry":"softmax","repo":"thaihungle/sm","repo_kind":"official","path":"utils.py","file_url":"https://github.com/thaihungle/sm/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"011537d0bea7fe09"}},{"code_sha256_prefix":"f2e3732f89410bea","entry":"unwrap","repo":"thaihungle/sm","repo_kind":"official","path":"envs.py","file_url":"https://github.com/thaihungle/sm/blob/HEAD/envs.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2e3732f89410bea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}