{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dual-policy-distillation","title":"Dual Policy Distillation","arxiv_id":"2006.04061","date":"2020-06-07","proceeding":null,"authors":["Kwei-Herng Lai","Daochen Zha","Yuening Li","Xia Hu"],"abstract":"Policy distillation, which transfers a teacher policy to a student policy has achieved great success in challenging tasks of deep reinforcement learning. This teacher-student framework requires a well-trained teacher model which is computationally expensive. Moreover, the performance of the student model could be limited by the teacher model if the teacher model is not optimal. In the light of collaborative learning, we study the feasibility of involving joint intellectual efforts from diverse perspectives of student models. In this work, we introduce dual policy distillation(DPD), a student-student framework in which two learners operate on the same environment to explore different perspectives of the environment and extract knowledge from each other to enhance their learning. The key challenge in developing this dual learning framework is to identify the beneficial knowledge from the peer learner for contemporary learning-based reinforcement learning algorithms, since it is unclear whether the knowledge distilled from an imperfect and noisy peer learner would be helpful. To address the challenge, we theoretically justify that distilling knowledge from a peer learner will lead to policy improvement and propose a disadvantageous distillation strategy based on the theoretical results. The conducted experiments on several continuous control tasks show that the proposed framework achieves superior performance with a learning-based agent and function approximation without the use of expensive teacher models.","url_abs":"https://arxiv.org/abs/2006.04061v1","url_pdf":"https://arxiv.org/pdf/2006.04061v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dual-policy-distillation","repo_url":"https://github.com/kiminh/dual-policy-distillation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.04061","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.04061"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kiminh/dual-policy-distillation","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/datamllab/dual-policy-distillation","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"summary":{"ran":1,"unverified":16},"by_repo_kind":{"listed":{"samples":17,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"98948fae5be7e92e","entry":"profile","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/logger.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"98948fae5be7e92e"}},{"code_sha256_prefix":"dafc66df201789e0","entry":"cat_entropy","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/a2c/utils.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"dafc66df201789e0"}},{"code_sha256_prefix":"7082b4bbc0d9384f","entry":"cat_entropy_softmax","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/a2c/utils.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"7082b4bbc0d9384f"}},{"code_sha256_prefix":"a14fa85826d8091c","entry":"cg","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/common/cg.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/common/cg.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a14fa85826d8091c"}},{"code_sha256_prefix":"71bdb8bae1ed81f7","entry":"find_task_for_env_id_in_any_benchmark","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/bench/benchmarks.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/bench/benchmarks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"71bdb8bae1ed81f7"}},{"code_sha256_prefix":"7bd934aa47da390a","entry":"get_benchmark","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/bench/benchmarks.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/bench/benchmarks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"7bd934aa47da390a"}},{"code_sha256_prefix":"c6cb2f61f176b29b","entry":"get_monitor_files","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/bench/monitor.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/bench/monitor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"c6cb2f61f176b29b"}},{"code_sha256_prefix":"3e82b9de30b1625f","entry":"get_task","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/bench/benchmarks.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/bench/benchmarks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"3e82b9de30b1625f"}},{"code_sha256_prefix":"e71620de1b94c6f0","entry":"load_results","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/bench/monitor.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/bench/monitor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"e71620de1b94c6f0"}},{"code_sha256_prefix":"186018cc4315055f","entry":"make_atari","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/common/atari_wrappers.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/common/atari_wrappers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"186018cc4315055f"}},{"code_sha256_prefix":"5eee517402b390d5","entry":"make_output_format","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/logger.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"5eee517402b390d5"}},{"code_sha256_prefix":"88ef5a17fc986d9a","entry":"read_json","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/logger.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"88ef5a17fc986d9a"}},{"code_sha256_prefix":"895ca4b776fb6d6d","entry":"rolling_window","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/results_plotter.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"895ca4b776fb6d6d"}},{"code_sha256_prefix":"178099b1f8fb2b9d","entry":"sample","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/a2c/utils.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"178099b1f8fb2b9d"}},{"code_sha256_prefix":"8c5b970d902942f8","entry":"ts2xy","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/results_plotter.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"8c5b970d902942f8"}},{"code_sha256_prefix":"43741b9339bd80da","entry":"window_func","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/results_plotter.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"43741b9339bd80da"}},{"code_sha256_prefix":"2475b5dbc5a1fb09","entry":"wrap_deepmind","repo":"kiminh/dual-policy-distillation","repo_kind":"listed","path":"baselines/common/atari_wrappers.py","file_url":"https://github.com/kiminh/dual-policy-distillation/blob/HEAD/baselines/common/atari_wrappers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"2475b5dbc5a1fb09"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}