{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multi-task-deep-reinforcement-learning-with","title":"Multi-task Deep Reinforcement Learning with PopArt","arxiv_id":"1809.04474","date":"2018-09-12","proceeding":null,"authors":["Matteo Hessel","Hubert Soyer","Lasse Espeholt","Wojciech Czarnecki","Simon Schmitt","Hado van Hasselt"],"abstract":"The reinforcement learning community has made great strides in designing\nalgorithms capable of exceeding human performance on specific tasks. These\nalgorithms are mostly trained one task at the time, each new task requiring to\ntrain a brand new agent instance. This means the learning algorithm is general,\nbut each solution is not; each agent can only solve the one task it was trained\non. In this work, we study the problem of learning to master not one but\nmultiple sequential-decision tasks at once. A general issue in multi-task\nlearning is that a balance must be found between the needs of multiple tasks\ncompeting for the limited resources of a single learning system. Many learning\nalgorithms can get distracted by certain tasks in the set of tasks to solve.\nSuch tasks appear more salient to the learning process, for instance because of\nthe density or magnitude of the in-task rewards. This causes the algorithm to\nfocus on those salient tasks at the expense of generality. We propose to\nautomatically adapt the contribution of each task to the agent's updates, so\nthat all tasks have a similar impact on the learning dynamics. This resulted in\nstate of the art performance on learning to play all games in a set of 57\ndiverse Atari games. Excitingly, our method learned a single trained policy -\nwith a single set of weights - that exceeds median human performance. To our\nknowledge, this was the first time a single agent surpassed human-level\nperformance on this multi-task domain. The same approach also demonstrated\nstate of the art performance on a set of 30 tasks in the 3D reinforcement\nlearning platform DeepMind Lab.","url_abs":"http://arxiv.org/abs/1809.04474v1","url_pdf":"http://arxiv.org/pdf/1809.04474v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multi-task-deep-reinforcement-learning-with","repo_url":"https://github.com/aluscher/torchbeastpopart","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"multi-task-deep-reinforcement-learning-with","repo_url":"https://github.com/opendilab/DI-engine","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-navigation-on-dmlab-30","task":"Visual Navigation","dataset":"Dmlab-30","model":"PopArt-IMPALA","rank_in_archive_order":1,"of":1,"metrics":{"Medium Human-Normalized Score":"72.8%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1809.04474","atlas_url":"https://app.syntology.ai/?focus=1809.04474","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1809.04474"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aluscher/torchbeastpopart","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opendilab/DI-engine","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"e9123707cba84b2c","entry":"compute_baseline_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"e9123707cba84b2c"}},{"code_sha256_prefix":"a27a9dc97eefda7f","entry":"compute_entropy_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a27a9dc97eefda7f"}},{"code_sha256_prefix":"fd36e71a3f45b446","entry":"compute_policy_gradient_loss","repo":"aluscher/torchbeastpopart","repo_kind":"listed","path":"torchbeast/polybeast.py","file_url":"https://github.com/aluscher/torchbeastpopart/blob/HEAD/torchbeast/polybeast.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd36e71a3f45b446"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}