{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-to-correctly-do-semantic-backpropagation","title":"How to Correctly do Semantic Backpropagation on Language-based Agentic Systems","arxiv_id":"2412.03624","date":"2024-12-04","proceeding":null,"authors":["Wenyi Wang","Hisham A. Alyahya","Dylan R. Ashley","Oleg Serikov","Dmitrii Khizbullin","Francesco Faccio","Jürgen Schmidhuber"],"abstract":"Language-based agentic systems have shown great promise in recent years, transitioning from solving small-scale research problems to being deployed in challenging real-world tasks. However, optimizing these systems often requires substantial manual labor. Recent studies have demonstrated that these systems can be represented as computational graphs, enabling automatic optimization. Despite these advancements, most current efforts in Graph-based Agentic System Optimization (GASO) fail to properly assign feedback to the system's components given feedback on the system's output. To address this challenge, we formalize the concept of semantic backpropagation with semantic gradients -- a generalization that aligns several key optimization techniques, including reverse-mode automatic differentiation and the more recent TextGrad by exploiting the relationship among nodes with a common successor. This serves as a method for computing directional information about how changes to each component of an agentic system might improve the system's output. To use these gradients, we propose a method called semantic gradient descent which enables us to solve GASO effectively. Our results on both BIG-Bench Hard and GSM8K show that our approach outperforms existing state-of-the-art methods for solving GASO problems. A detailed ablation study on the LIAR dataset demonstrates the parsimonious nature of our method. A full copy of our implementation is publicly available at https://github.com/HishamAlyahya/semantic_backprop","url_abs":"https://arxiv.org/abs/2412.03624v1","url_pdf":"https://arxiv.org/pdf/2412.03624v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-to-correctly-do-semantic-backpropagation","repo_url":"https://github.com/hishamalyahya/semantic_backprop","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"}],"methods":[{"method_slug":"textgrad","method_name":"TextGrad"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2412.03624","atlas_url":"https://app.syntology.ai/?focus=2412.03624","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.03624"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hishamalyahya/semantic_backprop","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0d63c9efe3b4310f","entry":"add_indent","repo":"hishamalyahya/semantic_backprop","repo_kind":"official","path":"semantic_backprop/utils.py","file_url":"https://github.com/hishamalyahya/semantic_backprop/blob/HEAD/semantic_backprop/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d63c9efe3b4310f"}},{"code_sha256_prefix":"3abb33733cf131f2","entry":"aprocess_example","repo":"hishamalyahya/semantic_backprop","repo_kind":"official","path":"semantic_backprop/engine/solutions.py","file_url":"https://github.com/hishamalyahya/semantic_backprop/blob/HEAD/semantic_backprop/engine/solutions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3abb33733cf131f2"}},{"code_sha256_prefix":"ce3d76ede3c858bf","entry":"arun_evaluate","repo":"hishamalyahya/semantic_backprop","repo_kind":"official","path":"semantic_backprop/engine/solutions.py","file_url":"https://github.com/hishamalyahya/semantic_backprop/blob/HEAD/semantic_backprop/engine/solutions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce3d76ede3c858bf"}},{"code_sha256_prefix":"4d97d0128f9547c9","entry":"fdbk_prompt","repo":"hishamalyahya/semantic_backprop","repo_kind":"official","path":"semantic_backprop/prompt_tmp.py","file_url":"https://github.com/hishamalyahya/semantic_backprop/blob/HEAD/semantic_backprop/prompt_tmp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d97d0128f9547c9"}},{"code_sha256_prefix":"410fde5d86d72617","entry":"fdbk_prompt_no_sibling","repo":"hishamalyahya/semantic_backprop","repo_kind":"official","path":"semantic_backprop/prompt_tmp.py","file_url":"https://github.com/hishamalyahya/semantic_backprop/blob/HEAD/semantic_backprop/prompt_tmp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"410fde5d86d72617"}},{"code_sha256_prefix":"4fd607b064edf51f","entry":"listing","repo":"hishamalyahya/semantic_backprop","repo_kind":"official","path":"semantic_backprop/utils.py","file_url":"https://github.com/hishamalyahya/semantic_backprop/blob/HEAD/semantic_backprop/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fd607b064edf51f"}},{"code_sha256_prefix":"9cded598fbc7acf1","entry":"parse_tagged_text","repo":"hishamalyahya/semantic_backprop","repo_kind":"official","path":"semantic_backprop/utils.py","file_url":"https://github.com/hishamalyahya/semantic_backprop/blob/HEAD/semantic_backprop/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cded598fbc7acf1"}},{"code_sha256_prefix":"5cd43eb4afc2e19e","entry":"sample_error_batch","repo":"hishamalyahya/semantic_backprop","repo_kind":"official","path":"semantic_backprop/engine/solutions.py","file_url":"https://github.com/hishamalyahya/semantic_backprop/blob/HEAD/semantic_backprop/engine/solutions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cd43eb4afc2e19e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}