{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reinforcement-learning-and-control-as","title":"Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review","arxiv_id":"1805.00909","date":"2018-05-02","proceeding":null,"authors":["Sergey Levine"],"abstract":"The framework of reinforcement learning or optimal control provides a\nmathematical formalization of intelligent decision making that is powerful and\nbroadly applicable. While the general form of the reinforcement learning\nproblem enables effective reasoning about uncertainty, the connection between\nreinforcement learning and inference in probabilistic models is not immediately\nobvious. However, such a connection has considerable value when it comes to\nalgorithm design: formalizing a problem as probabilistic inference in principle\nallows us to bring to bear a wide array of approximate inference tools, extend\nthe model in flexible and powerful ways, and reason about compositionality and\npartial observability. In this article, we will discuss how a generalization of\nthe reinforcement learning or optimal control problem, which is sometimes\ntermed maximum entropy reinforcement learning, is equivalent to exact\nprobabilistic inference in the case of deterministic dynamics, and variational\ninference in the case of stochastic dynamics. We will present a detailed\nderivation of this framework, overview prior work that has drawn on this and\nrelated ideas to propose new reinforcement learning and control algorithms, and\ndescribe perspectives on future research.","url_abs":"http://arxiv.org/abs/1805.00909v3","url_pdf":"http://arxiv.org/pdf/1805.00909v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reinforcement-learning-and-control-as","repo_url":"https://github.com/alex-petrenko/sample-factory","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"reinforcement-learning-and-control-as","repo_url":"https://github.com/sjoshi804/CS-267A-Final-Project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"variational-inference","task_name":"Variational Inference"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1805.00909","atlas_url":"https://app.syntology.ai/?focus=1805.00909","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1805.00909"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alex-petrenko/sample-factory","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sjoshi804/CS-267A-Final-Project","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c6a3e7a2b4cee020","entry":"allowed_actions","repo":"sjoshi804/CS-267A-Final-Project","repo_kind":"listed","path":"agent_pyro.py","file_url":"https://github.com/sjoshi804/CS-267A-Final-Project/blob/HEAD/agent_pyro.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6a3e7a2b4cee020"}},{"code_sha256_prefix":"fcd52224251960ce","entry":"reward_function","repo":"sjoshi804/CS-267A-Final-Project","repo_kind":"listed","path":"agent_pyro.py","file_url":"https://github.com/sjoshi804/CS-267A-Final-Project/blob/HEAD/agent_pyro.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcd52224251960ce"}},{"code_sha256_prefix":"2bdf3e7607ea5496","entry":"transition","repo":"sjoshi804/CS-267A-Final-Project","repo_kind":"listed","path":"agent_pyro.py","file_url":"https://github.com/sjoshi804/CS-267A-Final-Project/blob/HEAD/agent_pyro.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2bdf3e7607ea5496"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}