{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/counterfactual-conservative-q-learning-for-1","title":"Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning","arxiv_id":"2309.12696","date":"2023-09-22","proceeding":"NeurIPS 2023 11","authors":["Jianzhun Shao","Yun Qu","Chen Chen","Hongchang Zhang","Xiangyang Ji"],"abstract":"Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the action out-of-distribution (OOD) and value overestimation phenomenon excessively severe. Tomitigate this problem, we propose a novel multi-agent offline RL algorithm, named CounterFactual Conservative Q-Learning (CFCQL) to conduct conservative value estimation. Rather than regarding all the agents as a high dimensional single one and directly applying single agent methods to it, CFCQL calculates conservative regularization for each agent separately in a counterfactual way and then linearly combines them to realize an overall conservative value estimation. We prove that it still enjoys the underestimation property and the performance guarantee as those single agent conservative methods do, but the induced regularization and safe policy improvement bound are independent of the agent number, which is therefore theoretically superior to the direct treatment referred to above, especially when the agent number is large. We further conduct experiments on four environments including both discrete and continuous action settings on both existing and our man-made datasets, demonstrating that CFCQL outperforms existing methods on most datasets and even with a remarkable margin on some of them.","url_abs":"https://arxiv.org/abs/2309.12696v1","url_pdf":"https://arxiv.org/pdf/2309.12696v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"counterfactual-conservative-q-learning-for-1","repo_url":"https://github.com/thu-rllab/CFCQL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"multi-agent-reinforcement-learning","task_name":"Multi-agent Reinforcement Learning"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":null,"task_name":"counterfactual"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2309.12696","atlas_url":"https://app.syntology.ai/?focus=2309.12696","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.12696"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-rllab/CFCQL","reach":null}],"summary":{"ran":4,"ran_draft_wrong":3,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"4bf51e76634261e2","entry":"EpisodeBatch","repo":"thu-rllab/CFCQL","repo_kind":"official","path":"discrete/src/learners/cq_learner.py","file_url":"https://github.com/thu-rllab/CFCQL/blob/HEAD/discrete/src/learners/cq_learner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bf51e76634261e2"}},{"code_sha256_prefix":"007d9448d2cd2306","entry":"Mixer","repo":"thu-rllab/CFCQL","repo_kind":"official","path":"discrete/src/learners/cq_learner.py","file_url":"https://github.com/thu-rllab/CFCQL/blob/HEAD/discrete/src/learners/cq_learner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"007d9448d2cd2306"}},{"code_sha256_prefix":"a8b96233900018b1","entry":"QattenMixer","repo":"thu-rllab/CFCQL","repo_kind":"official","path":"discrete/src/learners/cq_learner.py","file_url":"https://github.com/thu-rllab/CFCQL/blob/HEAD/discrete/src/learners/cq_learner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8b96233900018b1"}},{"code_sha256_prefix":"876bdefdb02890de","entry":"VDNMixer","repo":"thu-rllab/CFCQL","repo_kind":"official","path":"discrete/src/learners/cq_learner.py","file_url":"https://github.com/thu-rllab/CFCQL/blob/HEAD/discrete/src/learners/cq_learner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"876bdefdb02890de"}},{"code_sha256_prefix":"a8e0e27ac8c6a7ec","entry":"build_q_lambda_targets","repo":"thu-rllab/CFCQL","repo_kind":"official","path":"discrete/src/learners/cq_learner.py","file_url":"https://github.com/thu-rllab/CFCQL/blob/HEAD/discrete/src/learners/cq_learner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8e0e27ac8c6a7ec"}},{"code_sha256_prefix":"d95214ebaa722aa0","entry":"build_td_lambda_targets","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d95214ebaa722aa0"}},{"code_sha256_prefix":"6511fdf3dee84c1c","entry":"get_parameters_num","repo":"thu-rllab/CFCQL","repo_kind":"official","path":"discrete/src/learners/cq_learner.py","file_url":"https://github.com/thu-rllab/CFCQL/blob/HEAD/discrete/src/learners/cq_learner.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6511fdf3dee84c1c"}},{"code_sha256_prefix":"fc990af09cc00be8","entry":"CQLearner","repo":"thu-rllab/CFCQL","repo_kind":"official","path":"discrete/src/learners/cq_learner.py","file_url":"https://github.com/thu-rllab/CFCQL/blob/HEAD/discrete/src/learners/cq_learner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc990af09cc00be8"}},{"code_sha256_prefix":"f27d16af7e01f025","entry":"orthogonal_init_","repo":"thu-rllab/CFCQL","repo_kind":"official","path":"discrete/src/learners/cq_learner.py","file_url":"https://github.com/thu-rllab/CFCQL/blob/HEAD/discrete/src/learners/cq_learner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f27d16af7e01f025"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}