{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mitigating-partial-observability-in","title":"Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy","arxiv_id":"2407.07333","date":"2024-07-10","proceeding":null,"authors":["Cameron Allen","Aaron Kirtland","Ruo Yu Tao","Sam Lobel","Daniel Scott","Nicholas Petrocelli","Omer Gottesman","Ronald Parr","Michael L. Littman","George Konidaris"],"abstract":"Reinforcement learning algorithms typically rely on the assumption that the environment dynamics and value function can be expressed in terms of a Markovian state representation. However, when state information is only partially observable, how can an agent learn such a state representation, and how can it detect when it has found one? We introduce a metric that can accomplish both objectives, without requiring access to -- or knowledge of -- an underlying, unobservable state space. Our metric, the $\\lambda$-discrepancy, is the difference between two distinct temporal difference (TD) value estimates, each computed using TD($\\lambda$) with a different value of $\\lambda$. Since TD($\\lambda{=}0$) makes an implicit Markov assumption and TD($\\lambda{=}1$) does not, a discrepancy between these estimates is a potential indicator of a non-Markovian state representation. Indeed, we prove that the $\\lambda$-discrepancy is exactly zero for all Markov decision processes and almost always non-zero for a broad class of partially observable environments. We also demonstrate empirically that, once detected, minimizing the $\\lambda$-discrepancy can help with learning a memory function to mitigate the corresponding partial observability. We then train a reinforcement learning agent that simultaneously constructs two recurrent value networks with different $\\lambda$ parameters and minimizes the difference between them as an auxiliary loss. The approach scales to challenging partially observable domains, where the resulting agent frequently performs significantly better (and never performs worse) than a baseline recurrent agent with only a single value network.","url_abs":"https://arxiv.org/abs/2407.07333v3","url_pdf":"https://arxiv.org/pdf/2407.07333v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mitigating-partial-observability-in","repo_url":"https://github.com/brownirl/lambda_discrepancy","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.07333","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.07333"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brownirl/lambda_discrepancy","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":8,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1af36889aa4d0d5a","entry":"filter_period_first_dim","repo":"brownirl/lambda_discrepancy","repo_kind":"official","path":"batch_run_ppo.py","file_url":"https://github.com/brownirl/lambda_discrepancy/blob/HEAD/batch_run_ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1af36889aa4d0d5a"}},{"code_sha256_prefix":"5992dc0c5426ce58","entry":"is_stochastic","repo":"brownirl/lambda_discrepancy","repo_kind":"official","path":"lamb/mdp.py","file_url":"https://github.com/brownirl/lambda_discrepancy/blob/HEAD/lamb/mdp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5992dc0c5426ce58"}},{"code_sha256_prefix":"b0113ccd72a50376","entry":"mse","repo":"brownirl/lambda_discrepancy","repo_kind":"official","path":"lamb/utils/loss.py","file_url":"https://github.com/brownirl/lambda_discrepancy/blob/HEAD/lamb/utils/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0113ccd72a50376"}},{"code_sha256_prefix":"d812c00f9ab45ac5","entry":"normalize","repo":"brownirl/lambda_discrepancy","repo_kind":"official","path":"lamb/mdp.py","file_url":"https://github.com/brownirl/lambda_discrepancy/blob/HEAD/lamb/mdp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d812c00f9ab45ac5"}},{"code_sha256_prefix":"dc3b6740211dcd35","entry":"place_ship_randomly","repo":"brownirl/lambda_discrepancy","repo_kind":"official","path":"lamb/envs/battleship.py","file_url":"https://github.com/brownirl/lambda_discrepancy/blob/HEAD/lamb/envs/battleship.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc3b6740211dcd35"}},{"code_sha256_prefix":"a0810715610cda8b","entry":"random_sparse_mask","repo":"brownirl/lambda_discrepancy","repo_kind":"official","path":"lamb/mdp.py","file_url":"https://github.com/brownirl/lambda_discrepancy/blob/HEAD/lamb/mdp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a0810715610cda8b"}},{"code_sha256_prefix":"7031ccd2469ccb81","entry":"seq_sarsa_loss","repo":"brownirl/lambda_discrepancy","repo_kind":"official","path":"lamb/utils/loss.py","file_url":"https://github.com/brownirl/lambda_discrepancy/blob/HEAD/lamb/utils/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7031ccd2469ccb81"}},{"code_sha256_prefix":"5fcd7a41eeee1cdf","entry":"seq_sarsa_mc_loss","repo":"brownirl/lambda_discrepancy","repo_kind":"official","path":"lamb/utils/loss.py","file_url":"https://github.com/brownirl/lambda_discrepancy/blob/HEAD/lamb/utils/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5fcd7a41eeee1cdf"}},{"code_sha256_prefix":"b95f9cd0b00f25b3","entry":"env_step","repo":"brownirl/lambda_discrepancy","repo_kind":"official","path":"lamb/agents/ppo.py","file_url":"https://github.com/brownirl/lambda_discrepancy/blob/HEAD/lamb/agents/ppo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b95f9cd0b00f25b3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}