{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-12604","title":"Beyond Softmax and Entropy: Convergence Rates of Policy Gradients with f-SoftArgmax Parameterization & Coupled Regularization","arxiv_id":"2601.12604","date":"2026-01-18","proceeding":null,"authors":["Safwan Labbi","Daniil Tiapkin","Paul Mangold","Eric Moulines"],"abstract":"Policy gradient methods are known to be highly sensitive to the choice of policy parameterization. In particular, the widely used softmax parameterization can induce ill-conditioned optimization landscapes and lead to exponentially slow convergence. Although this can be mitigated by preconditioning, this solution is often computationally expensive. Instead, we propose replacing the softmax with an alternative family of policy parameterizations based on the generalized f-softargmax. We further advocate coupling this parameterization with a regularizer induced by the same f-divergence, which improves the optimization landscape and ensures that the resulting regularized objective satisfies a Polyak-Lojasiewicz inequality. Leveraging this structure, we establish the first explicit non-asymptotic last-iterate convergence guarantees for stochastic policy gradient methods for finite MDPs without any form of preconditioning. We also derive sample-complexity bounds for the unregularized problem and show that f-PG, with Tsallis divergences achieves polynomial sample complexity in contrast to the exponential complexity incurred by the standard softmax parameterization.","url_abs":"https://arxiv.org/abs/2601.12604","url_pdf":"https://arxiv.org/pdf/2601.12604","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2601.12604","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.12604"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient","reach":{"status":"ok"}}],"summary":{"unverified":14},"by_repo_kind":{"found_in_text":{"samples":14,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"7595124d72d4181c","entry":"alpha_exp","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"DeepRL/fdiv.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/DeepRL/fdiv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7595124d72d4181c"}},{"code_sha256_prefix":"cc539c6fbe9a3990","entry":"alpha_exp","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"Tabular/fdiv.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/Tabular/fdiv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc539c6fbe9a3990"}},{"code_sha256_prefix":"92e5f5fb3e7984ed","entry":"alpha_log","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"DeepRL/fdiv.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/DeepRL/fdiv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92e5f5fb3e7984ed"}},{"code_sha256_prefix":"e5a462c891a4ae58","entry":"alpha_log","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"Tabular/fdiv.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/Tabular/fdiv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e5a462c891a4ae58"}},{"code_sha256_prefix":"7375e4f574cced94","entry":"build_groupings","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"DeepRL/plot_deepsea.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/DeepRL/plot_deepsea.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7375e4f574cced94"}},{"code_sha256_prefix":"ba9011862dbabe75","entry":"escort_policy_theta_row","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"Tabular/algos_jax.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/Tabular/algos_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba9011862dbabe75"}},{"code_sha256_prefix":"d724f48bf7125130","entry":"hadamard_policy_theta_row","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"Tabular/algos_jax.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/Tabular/algos_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d724f48bf7125130"}},{"code_sha256_prefix":"2585085c3f7d4f77","entry":"load_results_from_pickle","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"Tabular/plot_xp.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/Tabular/plot_xp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2585085c3f7d4f77"}},{"code_sha256_prefix":"57a6e47c0f84bf41","entry":"make_bisection","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"DeepRL/fdiv.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/DeepRL/fdiv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57a6e47c0f84bf41"}},{"code_sha256_prefix":"21284689471a4f92","entry":"parse_fpg","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"Tabular/plot_xp.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/Tabular/plot_xp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21284689471a4f92"}},{"code_sha256_prefix":"4e24a8f056d2aa1d","entry":"softmax_policy_theta_row","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"Tabular/algos_jax.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/Tabular/algos_jax.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e24a8f056d2aa1d"}},{"code_sha256_prefix":"d6e98275adcf3e98","entry":"subsample_series","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"DeepRL/plot_deepsea.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/DeepRL/plot_deepsea.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6e98275adcf3e98"}},{"code_sha256_prefix":"e499c81e7f18eeaa","entry":"subsample_series","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"Tabular/plot_xp.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/Tabular/plot_xp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e499c81e7f18eeaa"}},{"code_sha256_prefix":"759f1fbd2c22d004","entry":"summarize_runs","repo":"Labbi-Safwan/f-regularised-policy-gradient","repo_kind":"found_in_text","path":"DeepRL/plot_deepsea.py","file_url":"https://github.com/Labbi-Safwan/f-regularised-policy-gradient/blob/HEAD/DeepRL/plot_deepsea.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"759f1fbd2c22d004"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}