{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exploratory-combinatorial-optimization-with","title":"Exploratory Combinatorial Optimization with Reinforcement Learning","arxiv_id":"1909.04063","date":"2019-09-09","proceeding":null,"authors":["Thomas D. Barrett","William R. Clements","Jakob N. Foerster","A. I. Lvovsky"],"abstract":"Many real-world problems can be reduced to combinatorial optimization on a graph, where the subset or ordering of vertices that maximize some objective function must be found. With such tasks often NP-hard and analytically intractable, reinforcement learning (RL) has shown promise as a framework with which efficient heuristic methods to tackle these problems can be learned. Previous works construct the solution subset incrementally, adding one element at a time, however, the irreversible nature of this approach prevents the agent from revising its earlier decisions, which may be necessary given the complexity of the optimization task. We instead propose that the agent should seek to continuously improve the solution by learning to explore at test time. Our approach of exploratory combinatorial optimization (ECO-DQN) is, in principle, applicable to any combinatorial problem that can be defined on a graph. Experimentally, we show our method to produce state-of-the-art RL performance on the Maximum Cut problem. Moreover, because ECO-DQN can start from any arbitrary configuration, it can be combined with other search methods to further improve performance, which we demonstrate using a simple random search.","url_abs":"https://arxiv.org/abs/1909.04063v2","url_pdf":"https://arxiv.org/pdf/1909.04063v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exploratory-combinatorial-optimization-with","repo_url":"https://github.com/tomdbar/eco-dqn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"exploratory-combinatorial-optimization-with","repo_url":"https://github.com/ilBarbara/BioRL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"combinatorial-optimization","task_name":"Combinatorial Optimization"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1909.04063","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.04063"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ilBarbara/BioRL","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tomdbar/eco-dqn","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":20},"by_repo_kind":{"listed":{"samples":20,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ebb92c545dd37c5e","entry":"adj_dict_to_lists","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/mutation_data.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/mutation_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ebb92c545dd37c5e"}},{"code_sha256_prefix":"cd32e999ec66db43","entry":"bipartite_double_edge_swap","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/permute.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/permute.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd32e999ec66db43"}},{"code_sha256_prefix":"e3297cca27fd41f9","entry":"call_multidendrix","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/utils.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3297cca27fd41f9"}},{"code_sha256_prefix":"cd51a1fc23564cb6","entry":"check_condition","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/envs/saddlepoint.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/envs/saddlepoint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd51a1fc23564cb6"}},{"code_sha256_prefix":"0b789d75b024c791","entry":"choose_delta","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/output.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/output.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b789d75b024c791"}},{"code_sha256_prefix":"6fddb61386269dbe","entry":"compute_edge_dist","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/output.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/output.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6fddb61386269dbe"}},{"code_sha256_prefix":"d56306f4d27a55b3","entry":"construct_mutation_graph","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/permute.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/permute.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d56306f4d27a55b3"}},{"code_sha256_prefix":"b54eb9d11c17ff7b","entry":"convert_mutations_to_C_format","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/mutation_data.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/mutation_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b54eb9d11c17ff7b"}},{"code_sha256_prefix":"646d9fed47dadc7e","entry":"convert_solns","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/utils.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"646d9fed47dadc7e"}},{"code_sha256_prefix":"d3c7bc0639c710a4","entry":"discrete_convergence","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/convergence.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/convergence.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3c7bc0639c710a4"}},{"code_sha256_prefix":"3a84f2cd9eb58674","entry":"discrete_convergence_eqb_plot","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/convergence.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/convergence.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a84f2cd9eb58674"}},{"code_sha256_prefix":"ba02410b1835f606","entry":"enumeration","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/envs/saddlepoint.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/envs/saddlepoint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba02410b1835f606"}},{"code_sha256_prefix":"a388b8293614beeb","entry":"graph_to_mutation_data","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/permute.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/permute.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a388b8293614beeb"}},{"code_sha256_prefix":"204686180f489e85","entry":"initial_solns_generator","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/utils.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"204686180f489e85"}},{"code_sha256_prefix":"41514aa1cf823b3e","entry":"iter_num","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/output.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/output.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41514aa1cf823b3e"}},{"code_sha256_prefix":"159da52928b75420","entry":"load_graph","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"experiments/utils.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/experiments/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"159da52928b75420"}},{"code_sha256_prefix":"9887ce2dbe863145","entry":"load_graph_set","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"experiments/utils.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/experiments/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9887ce2dbe863145"}},{"code_sha256_prefix":"9e0d20c4942bc700","entry":"load_mutation_data","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/mutation_data.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/mutation_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e0d20c4942bc700"}},{"code_sha256_prefix":"f1deba67bc27ecef","entry":"saddlepoint","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/envs/saddlepoint.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/envs/saddlepoint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1deba67bc27ecef"}},{"code_sha256_prefix":"b28a2a9a7ba12564","entry":"tv","repo":"ilBarbara/BioRL","repo_kind":"listed","path":"src/comet/convergence.py","file_url":"https://github.com/ilBarbara/BioRL/blob/HEAD/src/comet/convergence.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b28a2a9a7ba12564"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}