{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/appbench-planning-of-multiple-apis-from","title":"AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction","arxiv_id":"2410.19743","date":"2024-10-10","proceeding":null,"authors":["Hongru Wang","Rui Wang","Boyang Xue","Heming Xia","Jingtao Cao","Zeming Liu","Jeff Z. Pan","Kam-Fai Wong"],"abstract":"Large Language Models (LLMs) can interact with the real world by connecting with versatile external APIs, resulting in better problem-solving and task automation capabilities. Previous research primarily focuses on APIs with limited arguments from a single source or overlooks the complex dependency relationship between different APIs. However, it is essential to utilize multiple APIs collaboratively from various sources (e.g., different Apps in the iPhone), especially for complex user instructions. In this paper, we introduce \\texttt{AppBench}, the first benchmark to evaluate LLMs' ability to plan and execute multiple APIs from various sources in order to complete the user's task. Specifically, we consider two significant challenges in multiple APIs: \\textit{1) graph structures:} some APIs can be executed independently while others need to be executed one by one, resulting in graph-like execution order; and \\textit{2) permission constraints:} which source is authorized to execute the API call. We have experimental results on 9 distinct LLMs; e.g., GPT-4o achieves only a 2.0\\% success rate at the most complex instruction, revealing that the existing state-of-the-art LLMs still cannot perform well in this situation even with the help of in-context learning and finetuning. Our code and data are publicly available at https://github.com/ruleGreen/AppBench.","url_abs":"https://arxiv.org/abs/2410.19743v1","url_pdf":"https://arxiv.org/pdf/2410.19743v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"appbench-planning-of-multiple-apis-from","repo_url":"https://github.com/ruleGreen/AppBench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.19743","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.19743"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ruleGreen/AppBench","reach":{"status":"ok"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"bd8890f7e40dc563","entry":"argus_approximate_match","repo":"ruleGreen/AppBench","repo_kind":"official","path":"code/appbench/analysis.py","file_url":"https://github.com/ruleGreen/AppBench/blob/HEAD/code/appbench/analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd8890f7e40dc563"}},{"code_sha256_prefix":"397544efb29f9ebc","entry":"compute_f1_direct","repo":"ruleGreen/AppBench","repo_kind":"official","path":"code/appbench/analysis.py","file_url":"https://github.com/ruleGreen/AppBench/blob/HEAD/code/appbench/analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"397544efb29f9ebc"}},{"code_sha256_prefix":"5450497d17869d94","entry":"extract_api_from_class","repo":"ruleGreen/AppBench","repo_kind":"official","path":"code/appbench/utils.py","file_url":"https://github.com/ruleGreen/AppBench/blob/HEAD/code/appbench/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5450497d17869d94"}},{"code_sha256_prefix":"bf4918acd17d5e92","entry":"extract_name_params","repo":"ruleGreen/AppBench","repo_kind":"official","path":"code/appbench/analysis.py","file_url":"https://github.com/ruleGreen/AppBench/blob/HEAD/code/appbench/analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf4918acd17d5e92"}},{"code_sha256_prefix":"29bd364b6b5e0cad","entry":"load_prompt_from_dir","repo":"ruleGreen/AppBench","repo_kind":"official","path":"code/appbench/utils.py","file_url":"https://github.com/ruleGreen/AppBench/blob/HEAD/code/appbench/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29bd364b6b5e0cad"}},{"code_sha256_prefix":"fdc4b6fb2e3064b8","entry":"read_functions","repo":"ruleGreen/AppBench","repo_kind":"official","path":"code/appbench/utils.py","file_url":"https://github.com/ruleGreen/AppBench/blob/HEAD/code/appbench/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fdc4b6fb2e3064b8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}