{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/magdi-structured-distillation-of-multi-agent","title":"MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models","arxiv_id":"2402.01620","date":"2024-02-02","proceeding":null,"authors":["Justin Chih-Yao Chen","Swarnadeep Saha","Elias Stengel-Eskin","Mohit Bansal"],"abstract":"Multi-agent interactions between Large Language Model (LLM) agents have shown major improvements on diverse reasoning tasks. However, these involve long generations from multiple models across several rounds, making them expensive. Moreover, these multi-agent approaches fail to provide a final, single model for efficient inference. To address this, we introduce MAGDi, a new method for structured distillation of the reasoning interactions between multiple LLMs into smaller LMs. MAGDi teaches smaller models by representing multi-agent interactions as graphs, augmenting a base student model with a graph encoder, and distilling knowledge using three objective functions: next-token prediction, a contrastive loss between correct and incorrect reasoning, and a graph-based objective to model the interaction structure. Experiments on seven widely used commonsense and math reasoning benchmarks show that MAGDi improves the reasoning capabilities of smaller models, outperforming several methods that distill from a single teacher and multiple teachers. Moreover, MAGDi also demonstrates an order of magnitude higher efficiency over its teachers. We conduct extensive analyses to show that MAGDi (1) enhances the generalizability to out-of-domain tasks, (2) scales positively with the size and strength of the base student model, and (3) obtains larger improvements (via our multi-teacher training) when applying self-consistency -- an inference technique that relies on model diversity.","url_abs":"https://arxiv.org/abs/2402.01620v2","url_pdf":"https://arxiv.org/pdf/2402.01620v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"magdi-structured-distillation-of-multi-agent","repo_url":"https://github.com/dinobby/magdi","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.01620","atlas_url":"https://app.syntology.ai/?focus=2402.01620","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.01620"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/dinobby/MAGDi","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dinobby/magdi","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bf01ec7c4053e257","entry":"construct_graphs","repo":"dinobby/MAGDi","repo_kind":"official","path":"utils.py","file_url":"https://github.com/dinobby/MAGDi/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf01ec7c4053e257"}},{"code_sha256_prefix":"b2178fcfdadace75","entry":"generate_ordered_list","repo":"dinobby/MAGDi","repo_kind":"official","path":"utils.py","file_url":"https://github.com/dinobby/MAGDi/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2178fcfdadace75"}},{"code_sha256_prefix":"6864bb6570ea95b0","entry":"is_equiv_1","repo":"dinobby/MAGDi","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/dinobby/MAGDi/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6864bb6570ea95b0"}},{"code_sha256_prefix":"c5f35db9cb688b48","entry":"normalize_fraction_notation","repo":"dinobby/MAGDi","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/dinobby/MAGDi/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5f35db9cb688b48"}},{"code_sha256_prefix":"9b09aa346afde94f","entry":"pad_graphs","repo":"dinobby/MAGDi","repo_kind":"official","path":"utils.py","file_url":"https://github.com/dinobby/MAGDi/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b09aa346afde94f"}},{"code_sha256_prefix":"6a272c4b18d461a9","entry":"prepare_test_data","repo":"dinobby/MAGDi","repo_kind":"official","path":"data_utils.py","file_url":"https://github.com/dinobby/MAGDi/blob/HEAD/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a272c4b18d461a9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}