{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/provably-efficient-maximum-entropy","title":"Provably Efficient Maximum Entropy Exploration","arxiv_id":"1812.02690","date":"2018-12-06","proceeding":null,"authors":["Elad Hazan","Sham M. Kakade","Karan Singh","Abby Van Soest"],"abstract":"Suppose an agent is in a (possibly unknown) Markov Decision Process in the\nabsence of a reward signal, what might we hope that an agent can efficiently\nlearn to do? This work studies a broad class of objectives that are defined\nsolely as functions of the state-visitation frequencies that are induced by how\nthe agent behaves. For example, one natural, intrinsically defined, objective\nproblem is for the agent to learn a policy which induces a distribution over\nstate space that is as uniform as possible, which can be measured in an\nentropic sense. We provide an efficient algorithm to optimize such such\nintrinsically defined objectives, when given access to a black box planning\noracle (which is robust to function approximation). Furthermore, when\nrestricted to the tabular setting where we have sample based access to the MDP,\nour proposed algorithm is provably efficient, both in terms of its sample and\ncomputational complexities. Key to our algorithmic methodology is utilizing the\nconditional gradient method (a.k.a. the Frank-Wolfe algorithm) which utilizes\nan approximate MDP solver.","url_abs":"http://arxiv.org/abs/1812.02690v2","url_pdf":"http://arxiv.org/pdf/1812.02690v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"provably-efficient-maximum-entropy","repo_url":"https://github.com/abbyvansoest/maxent_ant","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"provably-efficient-maximum-entropy","repo_url":"https://github.com/abbyvansoest/maxent","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1812.02690","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1812.02690"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abbyvansoest/maxent","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abbyvansoest/maxent_ant","reach":null}],"summary":{"ran_honours":1,"unverified":13},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1},"listed":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"f9a87d42f4aba379","entry":"discretize_range","repo":"abbyvansoest/maxent_ant","repo_kind":"official","path":"entropy/ant_utils.py","file_url":"https://github.com/abbyvansoest/maxent_ant/blob/HEAD/entropy/ant_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9a87d42f4aba379"}},{"code_sha256_prefix":"8627b1e582b17fad","entry":"discretize_range","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"base/base_utils.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/base/base_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8627b1e582b17fad"}},{"code_sha256_prefix":"e3d1de05bef0f98f","entry":"discretize_value","repo":"abbyvansoest/maxent_ant","repo_kind":"official","path":"entropy/ant_utils.py","file_url":"https://github.com/abbyvansoest/maxent_ant/blob/HEAD/entropy/ant_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3d1de05bef0f98f"}},{"code_sha256_prefix":"679710e830fe004c","entry":"fully_corrective_weights","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"679710e830fe004c"}},{"code_sha256_prefix":"3a04bcab9b7b15a2","entry":"geometric_weights","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a04bcab9b7b15a2"}},{"code_sha256_prefix":"cfce41fe4e302f84","entry":"get_all_datasets","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"plot.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/plot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfce41fe4e302f84"}},{"code_sha256_prefix":"ece9caed5d8cff07","entry":"get_datasets","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"plot.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/plot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ece9caed5d8cff07"}},{"code_sha256_prefix":"d8191ac2854bc916","entry":"get_next_file","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"plotting.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/plotting.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8191ac2854bc916"}},{"code_sha256_prefix":"01404759df85eaa8","entry":"get_num_states","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"base/base_utils.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/base/base_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01404759df85eaa8"}},{"code_sha256_prefix":"2b35e42973a59560","entry":"get_state","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"ant/ant_soft_actor_critic.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/ant/ant_soft_actor_critic.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b35e42973a59560"}},{"code_sha256_prefix":"a2580d7c4a763348","entry":"get_vars","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"core.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a2580d7c4a763348"}},{"code_sha256_prefix":"e68b07bcdded00fa","entry":"mlp","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"core.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e68b07bcdded00fa"}},{"code_sha256_prefix":"17b03c462ce92576","entry":"placeholder","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"core.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17b03c462ce92576"}},{"code_sha256_prefix":"25d36450d842d839","entry":"proj_unit_simplex","repo":"abbyvansoest/maxent","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/abbyvansoest/maxent/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25d36450d842d839"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}