{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/controlllm-augment-language-models-with-tools","title":"ControlLLM: Augment Language Models with Tools by Searching on Graphs","arxiv_id":"2310.17796","date":"2023-10-26","proceeding":null,"authors":["Zhaoyang Liu","Zeqiang Lai","Zhangwei Gao","Erfei Cui","Ziheng Li","Xizhou Zhu","Lewei Lu","Qifeng Chen","Yu Qiao","Jifeng Dai","Wenhai Wang"],"abstract":"We present ControlLLM, a novel framework that enables large language models (LLMs) to utilize multi-modal tools for solving complex real-world tasks. Despite the remarkable performance of LLMs, they still struggle with tool invocation due to ambiguous user prompts, inaccurate tool selection and parameterization, and inefficient tool scheduling. To overcome these challenges, our framework comprises three key components: (1) a \\textit{task decomposer} that breaks down a complex task into clear subtasks with well-defined inputs and outputs; (2) a \\textit{Thoughts-on-Graph (ToG) paradigm} that searches the optimal solution path on a pre-built tool graph, which specifies the parameter and dependency relations among different tools; and (3) an \\textit{execution engine with a rich toolbox} that interprets the solution path and runs the tools efficiently on different computational devices. We evaluate our framework on diverse tasks involving image, audio, and video processing, demonstrating its superior accuracy, efficiency, and versatility compared to existing methods. The code is at https://github.com/OpenGVLab/ControlLLM.","url_abs":"https://arxiv.org/abs/2310.17796v3","url_pdf":"https://arxiv.org/pdf/2310.17796v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"controlllm-augment-language-models-with-tools","repo_url":"https://github.com/opengvlab/controlllm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"scheduling","task_name":"Scheduling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.17796","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.17796"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/controlllm","reach":{"status":"ok"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"4e4ae4d35267d138","entry":"get_real_path","repo":"opengvlab/controlllm","repo_kind":"official","path":"cllm/utils.py","file_url":"https://github.com/opengvlab/controlllm/blob/HEAD/cllm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e4ae4d35267d138"}},{"code_sha256_prefix":"ff18914f07101301","entry":"md2plain","repo":"opengvlab/controlllm","repo_kind":"official","path":"cllm/utils.py","file_url":"https://github.com/opengvlab/controlllm/blob/HEAD/cllm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff18914f07101301"}},{"code_sha256_prefix":"e59e2d48825d9abd","entry":"plain2md","repo":"opengvlab/controlllm","repo_kind":"official","path":"cllm/utils.py","file_url":"https://github.com/opengvlab/controlllm/blob/HEAD/cllm/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e59e2d48825d9abd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}