{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-synthesize-programs-as","title":"Learning to Synthesize Programs as Interpretable and Generalizable Policies","arxiv_id":"2108.13643","date":"2021-08-31","proceeding":"NeurIPS 2021 12","authors":["Dweep Trivedi","Jesse Zhang","Shao-Hua Sun","Joseph J. Lim"],"abstract":"Recently, deep reinforcement learning (DRL) methods have achieved impressive performance on tasks in a variety of domains. However, neural network policies produced with DRL methods are not human-interpretable and often have difficulty generalizing to novel scenarios. To address these issues, prior works explore learning programmatic policies that are more interpretable and structured for generalization. Yet, these works either employ limited policy representations (e.g. decision trees, state machines, or predefined program templates) or require stronger supervision (e.g. input/output state pairs or expert demonstrations). We present a framework that instead learns to synthesize a program, which details the procedure to solve a task in a flexible and expressive manner, solely from reward signals. To alleviate the difficulty of learning to compose programs to induce the desired agent behavior from scratch, we propose to first learn a program embedding space that continuously parameterizes diverse behaviors in an unsupervised manner and then search over the learned program embedding space to yield a program that maximizes the return for a given task. Experimental results demonstrate that the proposed framework not only learns to reliably synthesize task-solving programs but also outperforms DRL and program synthesis baselines while producing interpretable and more generalizable policies. We also justify the necessity of the proposed two-stage learning scheme as well as analyze various methods for learning the program embedding.","url_abs":"https://arxiv.org/abs/2108.13643v4","url_pdf":"https://arxiv.org/pdf/2108.13643v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-synthesize-programs-as","repo_url":"https://github.com/clvrai/leaps","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"program-synthesis","task_name":"Program Synthesis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2108.13643","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2108.13643"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/clvrai/leaps","reach":null}],"summary":{"ran":6,"ran_draft_wrong":4,"ran_violates":2,"ran_fixture":3,"unverified":5},"by_repo_kind":{"listed":{"samples":20,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2070a867b9921946","entry":"CheckerState","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2070a867b9921946"}},{"code_sha256_prefix":"219771e4ff1f21bd","entry":"Encoder","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"219771e4ff1f21bd"}},{"code_sha256_prefix":"adff31c7886a3c8b","entry":"Karel_world_supervised","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"adff31c7886a3c8b"}},{"code_sha256_prefix":"57994492ff60cc1c","entry":"NNBase","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57994492ff60cc1c"}},{"code_sha256_prefix":"63334595c40e1b4f","entry":"PySyntaxChecker","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63334595c40e1b4f"}},{"code_sha256_prefix":"f109c2975e44ea7c","entry":"SyntaxVocabulary","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f109c2975e44ea7c"}},{"code_sha256_prefix":"68a7cd46f1488e51","entry":"_unmask_idx","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68a7cd46f1488e51"}},{"code_sha256_prefix":"709e9af7c826f886","entry":"_unmask_idx2","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"709e9af7c826f886"}},{"code_sha256_prefix":"4ec9bafe8f3946bb","entry":"front_is_clear","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ec9bafe8f3946bb"}},{"code_sha256_prefix":"552f40715e6a4741","entry":"get_location","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"552f40715e6a4741"}},{"code_sha256_prefix":"f169f731be71897b","entry":"get_neighbor","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f169f731be71897b"}},{"code_sha256_prefix":"6f0e0c50d69a79bf","entry":"masked_mean","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f0e0c50d69a79bf"}},{"code_sha256_prefix":"656bc963190be4b0","entry":"masked_sum","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"656bc963190be4b0"}},{"code_sha256_prefix":"d031ea99e1a1dd51","entry":"neighbor_is_clear","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d031ea99e1a1dd51"}},{"code_sha256_prefix":"b7fff67e10e92e79","entry":"state_transition","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7fff67e10e92e79"}},{"code_sha256_prefix":"9c0f14ab711773c9","entry":"ConditionPolicy","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c0f14ab711773c9"}},{"code_sha256_prefix":"692eaa2dd0184a17","entry":"Decoder","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"692eaa2dd0184a17"}},{"code_sha256_prefix":"49ce8132f623d5b6","entry":"ProgramVAE","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49ce8132f623d5b6"}},{"code_sha256_prefix":"5f52cac9a31533ed","entry":"VAE","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f52cac9a31533ed"}},{"code_sha256_prefix":"5e9899cd4f3029e3","entry":"check_type","repo":"clvrai/leaps","repo_kind":"listed","path":"pretrain/models.py","file_url":"https://github.com/clvrai/leaps/blob/HEAD/pretrain/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e9899cd4f3029e3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}