{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/offline-behavior-distillation","title":"Offline Behavior Distillation","arxiv_id":"2410.22728","date":"2024-10-30","proceeding":null,"authors":["Shiye Lei","Sen Zhang","DaCheng Tao"],"abstract":"Massive reinforcement learning (RL) data are typically collected to train policies offline without the need for interactions, but the large data volume can cause training inefficiencies. To tackle this issue, we formulate offline behavior distillation (OBD), which synthesizes limited expert behavioral data from sub-optimal RL data, enabling rapid policy learning. We propose two naive OBD objectives, DBC and PBC, which measure distillation performance via the decision difference between policies trained on distilled data and either offline data or a near-expert policy. Due to intractable bi-level optimization, the OBD objective is difficult to minimize to small values, which deteriorates PBC by its distillation performance guarantee with quadratic discount complexity $\\mathcal{O}(1/(1-\\gamma)^2)$. We theoretically establish the equivalence between the policy performance and action-value weighted decision difference, and introduce action-value weighted PBC (Av-PBC) as a more effective OBD objective. By optimizing the weighted decision difference, Av-PBC achieves a superior distillation guarantee with linear discount complexity $\\mathcal{O}(1/(1-\\gamma))$. Extensive experiments on multiple D4RL datasets reveal that Av-PBC offers significant improvements in OBD performance, fast distillation convergence speed, and robust cross-architecture/optimizer generalization.","url_abs":"https://arxiv.org/abs/2410.22728v1","url_pdf":"https://arxiv.org/pdf/2410.22728v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"offline-behavior-distillation","repo_url":"https://github.com/leaveslei/obd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.22728","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.22728"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LeavesLei/OBD","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leaveslei/obd","reach":{"status":"ok"}}],"summary":{"ran":1,"ran_draft_wrong":3,"ran_honours":3,"unverified":9},"by_repo_kind":{"official":{"samples":16,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"2adcff124b835b01","entry":"FullyConnectedQFunction","repo":"leaveslei/obd","repo_kind":"official","path":"data_lib/syndset.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/data_lib/syndset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2adcff124b835b01"}},{"code_sha256_prefix":"7b14e32ff9c72f54","entry":"_fanin_init","repo":"leaveslei/obd","repo_kind":"official","path":"data_lib/syndset.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/data_lib/syndset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b14e32ff9c72f54"}},{"code_sha256_prefix":"bd1f17e427bf51a5","entry":"dict2namespace","repo":"leaveslei/obd","repo_kind":"official","path":"obd_bptt.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/obd_bptt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd1f17e427bf51a5"}},{"code_sha256_prefix":"3155dcd1f7d26b1f","entry":"flatten","repo":"leaveslei/obd","repo_kind":"official","path":"obd_bptt.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/obd_bptt.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3155dcd1f7d26b1f"}},{"code_sha256_prefix":"bb104479d26e39c3","entry":"get_hms","repo":"LeavesLei/OBD","repo_kind":"official","path":"utils.py","file_url":"https://github.com/LeavesLei/OBD/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb104479d26e39c3"}},{"code_sha256_prefix":"d35eb4a0d5666e7f","entry":"merge_args","repo":"leaveslei/obd","repo_kind":"official","path":"configs/command_parser.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/configs/command_parser.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d35eb4a0d5666e7f"}},{"code_sha256_prefix":"ba80f9ff81b1de5f","entry":"normalize_states","repo":"LeavesLei/OBD","repo_kind":"official","path":"utils.py","file_url":"https://github.com/LeavesLei/OBD/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba80f9ff81b1de5f"}},{"code_sha256_prefix":"c0927665fe7f7dc7","entry":"Net","repo":"leaveslei/obd","repo_kind":"official","path":"data_lib/syndset.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/data_lib/syndset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0927665fe7f7dc7"}},{"code_sha256_prefix":"d540ae0c1a5c67bc","entry":"_constant_bias_init","repo":"leaveslei/obd","repo_kind":"official","path":"data_lib/syndset.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/data_lib/syndset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d540ae0c1a5c67bc"}},{"code_sha256_prefix":"30ac7cd064d9ecda","entry":"_orthogonal_init","repo":"leaveslei/obd","repo_kind":"official","path":"data_lib/syndset.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/data_lib/syndset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30ac7cd064d9ecda"}},{"code_sha256_prefix":"defae8a6ac156ec3","entry":"get_data","repo":"LeavesLei/OBD","repo_kind":"official","path":"data_lib/dset.py","file_url":"https://github.com/LeavesLei/OBD/blob/HEAD/data_lib/dset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"defae8a6ac156ec3"}},{"code_sha256_prefix":"c481fbf6b3811329","entry":"init_module_weights","repo":"leaveslei/obd","repo_kind":"official","path":"data_lib/syndset.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/data_lib/syndset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c481fbf6b3811329"}},{"code_sha256_prefix":"f297771ebb6401b8","entry":"layer_init","repo":"leaveslei/obd","repo_kind":"official","path":"data_lib/syndset.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/data_lib/syndset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f297771ebb6401b8"}},{"code_sha256_prefix":"dc43b3f4e178b8c1","entry":"merge_args","repo":"LeavesLei/OBD","repo_kind":"official","path":"configs/command_parser.py","file_url":"https://github.com/LeavesLei/OBD/blob/HEAD/configs/command_parser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc43b3f4e178b8c1"}},{"code_sha256_prefix":"1ec0ea84d63a7360","entry":"orthogonal_init","repo":"leaveslei/obd","repo_kind":"official","path":"data_lib/syndset.py","file_url":"https://github.com/leaveslei/obd/blob/HEAD/data_lib/syndset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1ec0ea84d63a7360"}},{"code_sha256_prefix":"6f4380c7d15b6e42","entry":"wrap_env","repo":"LeavesLei/OBD","repo_kind":"official","path":"utils.py","file_url":"https://github.com/LeavesLei/OBD/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f4380c7d15b6e42"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}