{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arsm-augment-reinforce-swap-merge-estimator","title":"ARSM: Augment-REINFORCE-Swap-Merge Estimator for Gradient Backpropagation Through Categorical Variables","arxiv_id":"1905.01413","date":"2019-05-04","proceeding":null,"authors":["Mingzhang Yin","Yuguang Yue","Mingyuan Zhou"],"abstract":"To address the challenge of backpropagating the gradient through categorical variables, we propose the augment-REINFORCE-swap-merge (ARSM) gradient estimator that is unbiased and has low variance. ARSM first uses variable augmentation, REINFORCE, and Rao-Blackwellization to re-express the gradient as an expectation under the Dirichlet distribution, then uses variable swapping to construct differently expressed but equivalent expectations, and finally shares common random numbers between these expectations to achieve significant variance reduction. Experimental results show ARSM closely resembles the performance of the true gradient for optimization in univariate settings; outperforms existing estimators by a large margin when applied to categorical variational auto-encoders; and provides a \"try-and-see self-critic\" variance reduction method for discrete-action policy gradient, which removes the need of estimating baselines by generating a random number of pseudo actions and estimating their action-value functions.","url_abs":"https://arxiv.org/abs/1905.01413v2","url_pdf":"https://arxiv.org/pdf/1905.01413v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"arsm-augment-reinforce-swap-merge-estimator","repo_url":"https://github.com/ARM-gradient/ARSM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[],"methods":[{"method_slug":"reinforce","method_name":"REINFORCE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1905.01413","atlas_url":"https://app.syntology.ai/?focus=1905.01413","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1905.01413"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ARM-gradient/ARSM","reach":null}],"summary":{"ran_violates":2,"ran_fixture":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"404304e71e2184c1","entry":"fun","repo":"ARM-gradient/ARSM","repo_kind":"official","path":"toy/ARSM_Univariate.py","file_url":"https://github.com/ARM-gradient/ARSM/blob/HEAD/toy/ARSM_Univariate.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"404304e71e2184c1"}},{"code_sha256_prefix":"673a92d176d30b76","entry":"pseudo_action_swap_matrix","repo":"ARM-gradient/ARSM","repo_kind":"official","path":"toy/ARSM_Univariate.py","file_url":"https://github.com/ARM-gradient/ARSM/blob/HEAD/toy/ARSM_Univariate.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"673a92d176d30b76"}},{"code_sha256_prefix":"ad039f77ef1e89fc","entry":"softmax","repo":"ARM-gradient/ARSM","repo_kind":"official","path":"toy/ARSM_Univariate.py","file_url":"https://github.com/ARM-gradient/ARSM/blob/HEAD/toy/ARSM_Univariate.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad039f77ef1e89fc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}