{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-simulate-self-driven-particles","title":"Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization","arxiv_id":"2110.13827","date":"2021-10-26","proceeding":"NeurIPS 2021 12","authors":["Zhenghao Peng","Quanyi Li","Ka Ming Hui","Chunxiao Liu","Bolei Zhou"],"abstract":"Self-Driven Particles (SDP) describe a category of multi-agent systems common in everyday life, such as flocking birds and traffic flows. In a SDP system, each agent pursues its own goal and constantly changes its cooperative or competitive behaviors with its nearby agents. Manually designing the controllers for such SDP system is time-consuming, while the resulting emergent behaviors are often not realistic nor generalizable. Thus the realistic simulation of SDP systems remains challenging. Reinforcement learning provides an appealing alternative for automating the development of the controller for SDP. However, previous multi-agent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand, which fail to capture the essence of SDP where the role of each agent varies to be cooperative or competitive even within one episode. To simulate SDP with MARL, a key challenge is to coordinate agents' behaviors while still maximizing individual objectives. Taking traffic simulation as the testing bed, in this work we develop a novel MARL method called Coordinated Policy Optimization (CoPO), which incorporates social psychology principle to learn neural controller for SDP. Experiments show that the proposed method can achieve superior performance compared to MARL baselines in various metrics. Noticeably the trained vehicles exhibit complex and diverse social behaviors that improve performance and safety of the population as a whole. Demo video and source code are available at: https://decisionforce.github.io/CoPO/","url_abs":"https://arxiv.org/abs/2110.13827v2","url_pdf":"https://arxiv.org/pdf/2110.13827v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-simulate-self-driven-particles","repo_url":"https://github.com/decisionforce/CoPO","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"multi-agent-reinforcement-learning","task_name":"Multi-agent Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.13827","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.13827"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/decisionforce/CoPO","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7f7ae3a2bf44fe29","entry":"build_meta_gradient","repo":"decisionforce/CoPO","repo_kind":"official","path":"copo_code/copo/algo_copo/grad.py","file_url":"https://github.com/decisionforce/CoPO/blob/HEAD/copo_code/copo/algo_copo/grad.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f7ae3a2bf44fe29"}},{"code_sha256_prefix":"01d145142200fe2c","entry":"compute_global_advantage","repo":"decisionforce/CoPO","repo_kind":"official","path":"copo_code/copo/algo_copo/copo.py","file_url":"https://github.com/decisionforce/CoPO/blob/HEAD/copo_code/copo/algo_copo/copo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01d145142200fe2c"}},{"code_sha256_prefix":"65c8249d273888dc","entry":"compute_nei_advantage","repo":"decisionforce/CoPO","repo_kind":"official","path":"copo_code/copo/algo_copo/copo.py","file_url":"https://github.com/decisionforce/CoPO/blob/HEAD/copo_code/copo/algo_copo/copo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65c8249d273888dc"}},{"code_sha256_prefix":"aaebe5145b957a38","entry":"concat_ccppo_process","repo":"decisionforce/CoPO","repo_kind":"official","path":"copo_code/copo/algo_ccppo/ccppo.py","file_url":"https://github.com/decisionforce/CoPO/blob/HEAD/copo_code/copo/algo_ccppo/ccppo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aaebe5145b957a38"}},{"code_sha256_prefix":"8c188ecb1900bbd2","entry":"get_change_n_env","repo":"decisionforce/CoPO","repo_kind":"official","path":"copo_code/copo/algo_ippo/ippo_cl.py","file_url":"https://github.com/decisionforce/CoPO/blob/HEAD/copo_code/copo/algo_ippo/ippo_cl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c188ecb1900bbd2"}},{"code_sha256_prefix":"c22001586dc8903a","entry":"mean_field_ccppo_process","repo":"decisionforce/CoPO","repo_kind":"official","path":"copo_code/copo/algo_ccppo/ccppo.py","file_url":"https://github.com/decisionforce/CoPO/blob/HEAD/copo_code/copo/algo_ccppo/ccppo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c22001586dc8903a"}},{"code_sha256_prefix":"ccdd46a7f409fd44","entry":"postprocess_add_advantages","repo":"decisionforce/CoPO","repo_kind":"official","path":"copo_code/copo/algo_copo/copo.py","file_url":"https://github.com/decisionforce/CoPO/blob/HEAD/copo_code/copo/algo_copo/copo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ccdd46a7f409fd44"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}