{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-imitation-learning-via-generalized-lower","title":"Self-Imitation Learning via Generalized Lower Bound Q-learning","arxiv_id":"2006.07442","date":"2020-06-12","proceeding":"NeurIPS 2020 12","authors":["Yunhao Tang"],"abstract":"Self-imitation learning motivated by lower-bound Q-learning is a novel and effective approach for off-policy learning. In this work, we propose a n-step lower bound which generalizes the original return-based lower-bound Q-learning, and introduce a new family of self-imitation learning algorithms. To provide a formal motivation for the potential performance gains provided by self-imitation learning, we show that n-step lower bound Q-learning achieves a trade-off between fixed point bias and contraction rate, drawing close connections to the popular uncorrected n-step Q-learning. We finally show that n-step lower bound Q-learning is a more robust alternative to return-based self-imitation learning and uncorrected n-step, over a wide range of continuous control benchmark tasks.","url_abs":"https://arxiv.org/abs/2006.07442v3","url_pdf":"https://arxiv.org/pdf/2006.07442v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2006.07442","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.07442"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/openai/spinningup","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/junhyukoh/self-imitation-learning","reach":null}],"summary":{"ran":11,"ran_draft_wrong":3,"ran_violates":2,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":20,"ran":16,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ae840f19840ded1e","entry":"MinSegmentTree","repo":"junhyukoh/self-imitation-learning","repo_kind":"found_in_text","path":"baselines/common/self_imitation.py","file_url":"https://github.com/junhyukoh/self-imitation-learning/blob/HEAD/baselines/common/self_imitation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae840f19840ded1e"}},{"code_sha256_prefix":"92fcff634a9f6c9e","entry":"ReplayBuffer","repo":"junhyukoh/self-imitation-learning","repo_kind":"found_in_text","path":"baselines/common/self_imitation.py","file_url":"https://github.com/junhyukoh/self-imitation-learning/blob/HEAD/baselines/common/self_imitation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92fcff634a9f6c9e"}},{"code_sha256_prefix":"a8ec19e70d73f9c2","entry":"SegmentTree","repo":"junhyukoh/self-imitation-learning","repo_kind":"found_in_text","path":"baselines/common/self_imitation.py","file_url":"https://github.com/junhyukoh/self-imitation-learning/blob/HEAD/baselines/common/self_imitation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8ec19e70d73f9c2"}},{"code_sha256_prefix":"7d650bc8cab9fa52","entry":"SumSegmentTree","repo":"junhyukoh/self-imitation-learning","repo_kind":"found_in_text","path":"baselines/common/self_imitation.py","file_url":"https://github.com/junhyukoh/self-imitation-learning/blob/HEAD/baselines/common/self_imitation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d650bc8cab9fa52"}},{"code_sha256_prefix":"2103e65a90ce7a04","entry":"all_bools","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/utils/run_utils.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/utils/run_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2103e65a90ce7a04"}},{"code_sha256_prefix":"7e36f1c357c4618d","entry":"colorize","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/utils/logx.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/utils/logx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e36f1c357c4618d"}},{"code_sha256_prefix":"57542f87e248db45","entry":"combined_shape","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/algos/pytorch/ddpg/core.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/algos/pytorch/ddpg/core.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57542f87e248db45"}},{"code_sha256_prefix":"fbc505a5dac82646","entry":"convert_json","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/utils/serialization_utils.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/utils/serialization_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbc505a5dac82646"}},{"code_sha256_prefix":"4cca5e460067b900","entry":"count_vars","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/algos/pytorch/ddpg/core.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/algos/pytorch/ddpg/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cca5e460067b900"}},{"code_sha256_prefix":"874462e54e7cc379","entry":"discount_with_dones","repo":"junhyukoh/self-imitation-learning","repo_kind":"found_in_text","path":"baselines/common/self_imitation.py","file_url":"https://github.com/junhyukoh/self-imitation-learning/blob/HEAD/baselines/common/self_imitation.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"874462e54e7cc379"}},{"code_sha256_prefix":"f29202d8108bccd3","entry":"flat_concat","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/utils/mpi_tf.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/utils/mpi_tf.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f29202d8108bccd3"}},{"code_sha256_prefix":"62314e3b8ca6edfc","entry":"get_all_datasets","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/utils/plot.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/utils/plot.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62314e3b8ca6edfc"}},{"code_sha256_prefix":"e642bb98eca55069","entry":"get_datasets","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/utils/plot.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/utils/plot.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e642bb98eca55069"}},{"code_sha256_prefix":"9ca8b8e45f7fbebf","entry":"is_json_serializable","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/utils/serialization_utils.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/utils/serialization_utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ca8b8e45f7fbebf"}},{"code_sha256_prefix":"7569e2a512ec417c","entry":"mlp","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/algos/pytorch/ddpg/core.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/algos/pytorch/ddpg/core.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7569e2a512ec417c"}},{"code_sha256_prefix":"c6bdd181afa1fdf7","entry":"valid_str","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/utils/run_utils.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/utils/run_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6bdd181afa1fdf7"}},{"code_sha256_prefix":"3d40ed2237ec8f20","entry":"PrioritizedReplayBuffer","repo":"junhyukoh/self-imitation-learning","repo_kind":"found_in_text","path":"baselines/common/self_imitation.py","file_url":"https://github.com/junhyukoh/self-imitation-learning/blob/HEAD/baselines/common/self_imitation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d40ed2237ec8f20"}},{"code_sha256_prefix":"1a23890f034cf35a","entry":"SelfImitation","repo":"junhyukoh/self-imitation-learning","repo_kind":"found_in_text","path":"baselines/common/self_imitation.py","file_url":"https://github.com/junhyukoh/self-imitation-learning/blob/HEAD/baselines/common/self_imitation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a23890f034cf35a"}},{"code_sha256_prefix":"042875cf711accd0","entry":"assign_params_from_flat","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/utils/mpi_tf.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/utils/mpi_tf.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"042875cf711accd0"}},{"code_sha256_prefix":"8a70bbd1276e274c","entry":"restore_tf_graph","repo":"openai/spinningup","repo_kind":"found_in_text","path":"spinup/utils/logx.py","file_url":"https://github.com/openai/spinningup/blob/HEAD/spinup/utils/logx.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a70bbd1276e274c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}