{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/backprop-q-generalized-backpropagation-for","title":"Backprop-Q: Generalized Backpropagation for Stochastic Computation Graphs","arxiv_id":"1807.09511","date":"2018-07-25","proceeding":null,"authors":["Xiaoran Xu","Songpeng Zu","Yuan Zhang","Hanning Zhou","Wei Feng"],"abstract":"In real-world scenarios, it is appealing to learn a model carrying out\nstochastic operations internally, known as stochastic computation graphs\n(SCGs), rather than learning a deterministic mapping. However, standard\nbackpropagation is not applicable to SCGs. We attempt to address this issue\nfrom the angle of cost propagation, with local surrogate costs, called\nQ-functions, constructed and learned for each stochastic node in an SCG. Then,\nthe SCG can be trained based on these surrogate costs using standard\nbackpropagation. We propose the entire framework as a solution to generalize\nbackpropagation for SCGs, which resembles an actor-critic architecture but\nbased on a graph. For broad applicability, we study a variety of SCG structures\nfrom one cost to multiple costs. We utilize recent advances in reinforcement\nlearning (RL) and variational Bayes (VB), such as off-policy critic learning\nand unbiased-and-low-variance gradient estimation, and review them in the\ncontext of SCGs. The generalized backpropagation extends transported learning\nsignals beyond gradients between stochastic nodes while preserving the benefit\nof backpropagating gradients through deterministic nodes. Experimental\nsuggestions and concerns are listed to help design and test any specific model\nusing this framework.","url_abs":"http://arxiv.org/abs/1807.09511v2","url_pdf":"http://arxiv.org/pdf/1807.09511v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"backprop-q-generalized-backpropagation-for","repo_url":"https://github.com/netpaladinx/experiments-for-Backprop-Q","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1807.09511","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}