{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-alternative-softmax-operator-for","title":"An Alternative Softmax Operator for Reinforcement Learning","arxiv_id":"1612.05628","date":"2016-12-16","proceeding":"ICML 2017 8","authors":["Kavosh Asadi","Michael L. Littman"],"abstract":"A softmax operator applied to a set of values acts somewhat like the\nmaximization function and somewhat like an average. In sequential decision\nmaking, softmax is often used in settings where it is necessary to maximize\nutility but also to hedge against problems that arise from putting all of one's\nweight behind a single maximum utility decision. The Boltzmann softmax operator\nis the most commonly used softmax operator in this setting, but we show that\nthis operator is prone to misbehavior. In this work, we study a differentiable\nsoftmax operator that, among other properties, is a non-expansion ensuring a\nconvergent behavior in learning and planning. We introduce a variant of SARSA\nalgorithm that, by utilizing the new operator, computes a Boltzmann policy with\na state-dependent temperature parameter. We show that the algorithm is\nconvergent and that it performs favorably in practice.","url_abs":"http://arxiv.org/abs/1612.05628v5","url_pdf":"http://arxiv.org/pdf/1612.05628v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-alternative-softmax-operator-for","repo_url":"https://github.com/grayswanai/nanogcg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"an-alternative-softmax-operator-for","repo_url":"https://github.com/sudh1976/ECE510_DeepLearning_Project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"an-alternative-softmax-operator-for","repo_url":"https://github.com/thompsonj/nanoGCG","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"sequential-decision-making","task_name":"Sequential Decision Making"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1612.05628","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}