{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/nestful-a-benchmark-for-evaluating-llms-on","title":"NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls","arxiv_id":"2409.03797","date":"2024-09-04","proceeding":null,"authors":["Kinjal Basu","Ibrahim Abdelaziz","Kiran Kate","Mayank Agarwal","Maxwell Crouse","Yara Rizk","Kelsey Bradford","Asim Munawar","Sadhana Kumaravel","Saurabh Goyal","Xin Wang","Luis A. Lastras","Pavan Kapanipathi"],"abstract":"The resurgence of autonomous agents built using large language models (LLMs) to solve complex real-world tasks has brought increased focus on LLMs' fundamental ability of tool or function calling. At the core of these agents, an LLM must plan, execute, and respond using external tools, APIs, and custom functions. Research on tool calling has gathered momentum, but evaluation benchmarks and datasets representing the complexity of the tasks have lagged behind. In this work, we focus on one such complexity, nested sequencing, with the goal of extending existing benchmarks and evaluation. Specifically, we present NESTFUL, a benchmark to evaluate LLMs on nested sequences of API calls, i.e., sequences where the output of one API call is passed as input to a subsequent call. NESTFUL contains 1800+ nested sequences where all the function calls are executable. Experimental results on a variety of models show that the best-performing model (GPT-4o) achieves a full sequence match accuracy of 28% and a win-rate of 60%, necessitating a large scope for improvement in the nested sequencing aspect of function calling. Our analysis of these results provides possible future research directions for the community, in addition to a benchmark to track progress. We have released the NESTFUL dataset under the Apache 2.0 license at https://github.com/IBM/NESTFUL.","url_abs":"https://arxiv.org/abs/2409.03797v3","url_pdf":"https://arxiv.org/pdf/2409.03797v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"nestful-a-benchmark-for-evaluating-llms-on","repo_url":"https://github.com/ibm/nestful","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.03797","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.03797"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ibm/nestful","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":17,"unverified":4},"by_repo_kind":{"official":{"samples":21,"ran":17,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9c6d8f0b3847114f","entry":"ast_parse","repo":"ibm/nestful","repo_kind":"official","path":"src/ast_utils.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/ast_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c6d8f0b3847114f"}},{"code_sha256_prefix":"ce18037f4b6348c6","entry":"create_service_account","repo":"ibm/nestful","repo_kind":"official","path":"data_v2/executable_functions/py_code_file_1001.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/data_v2/executable_functions/py_code_file_1001.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce18037f4b6348c6"}},{"code_sha256_prefix":"153e1cc7a8f8d341","entry":"do_stuff","repo":"ibm/nestful","repo_kind":"official","path":"data_v2/executable_functions/py_code_file_1002.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/data_v2/executable_functions/py_code_file_1002.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"153e1cc7a8f8d341"}},{"code_sha256_prefix":"31c2e27c79774797","entry":"extend_num","repo":"ibm/nestful","repo_kind":"official","path":"data_v2/executable_functions/py_code_file_100.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/data_v2/executable_functions/py_code_file_100.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"31c2e27c79774797"}},{"code_sha256_prefix":"47cc520d6946b782","entry":"find_strings_with_pattern","repo":"ibm/nestful","repo_kind":"official","path":"data_v2/executable_functions/py_code_file_1.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/data_v2/executable_functions/py_code_file_1.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47cc520d6946b782"}},{"code_sha256_prefix":"fb5af146b3521b50","entry":"format_padding","repo":"ibm/nestful","repo_kind":"official","path":"data_v2/executable_functions/py_code_file_10.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/data_v2/executable_functions/py_code_file_10.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb5af146b3521b50"}},{"code_sha256_prefix":"a58b18082fe1da3f","entry":"get_deli_sep_str_list","repo":"ibm/nestful","repo_kind":"official","path":"src/output_parsers.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/output_parsers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a58b18082fe1da3f"}},{"code_sha256_prefix":"48207fba685d16d6","entry":"get_icl_str","repo":"ibm/nestful","repo_kind":"official","path":"src/instruct_data_prep.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/instruct_data_prep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"48207fba685d16d6"}},{"code_sha256_prefix":"f0a4f5a7f292a3d0","entry":"listit","repo":"ibm/nestful","repo_kind":"official","path":"src/scorer.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/scorer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0a4f5a7f292a3d0"}},{"code_sha256_prefix":"b524a104e61bdc60","entry":"permutation","repo":"ibm/nestful","repo_kind":"official","path":"data_v2/executable_functions/basic_functions.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/data_v2/executable_functions/basic_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b524a104e61bdc60"}},{"code_sha256_prefix":"29da7d03953ecf87","entry":"power","repo":"ibm/nestful","repo_kind":"official","path":"data_v2/executable_functions/basic_functions.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/data_v2/executable_functions/basic_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29da7d03953ecf87"}},{"code_sha256_prefix":"cd5994aa5f8341d1","entry":"process_slot_value","repo":"ibm/nestful","repo_kind":"official","path":"src/output_parsers.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/output_parsers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cd5994aa5f8341d1"}},{"code_sha256_prefix":"ba6cfefa8094dcb6","entry":"read_file","repo":"ibm/nestful","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba6cfefa8094dcb6"}},{"code_sha256_prefix":"cc91f822de63292f","entry":"resolve_ast_by_type","repo":"ibm/nestful","repo_kind":"official","path":"src/ast_utils.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/ast_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc91f822de63292f"}},{"code_sha256_prefix":"053e58f476c2e09c","entry":"resolve_ast_call","repo":"ibm/nestful","repo_kind":"official","path":"src/ast_utils.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/ast_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"053e58f476c2e09c"}},{"code_sha256_prefix":"662385c90863ea91","entry":"simplify_slug","repo":"ibm/nestful","repo_kind":"official","path":"data_v2/executable_functions/py_code_file_1000.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/data_v2/executable_functions/py_code_file_1000.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"662385c90863ea91"}},{"code_sha256_prefix":"7f2f2e2df15e6716","entry":"subtract","repo":"ibm/nestful","repo_kind":"official","path":"data_v2/executable_functions/basic_functions.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/data_v2/executable_functions/basic_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f2f2e2df15e6716"}},{"code_sha256_prefix":"f5b3c135f40d531c","entry":"calculate_ans","repo":"ibm/nestful","repo_kind":"official","path":"src/scorer.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/scorer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f5b3c135f40d531c"}},{"code_sha256_prefix":"c9fb850a9577a4d4","entry":"get_basic_func_list","repo":"ibm/nestful","repo_kind":"official","path":"src/scorer.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/scorer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9fb850a9577a4d4"}},{"code_sha256_prefix":"63f870a98c11420f","entry":"get_granite_tokenizer","repo":"ibm/nestful","repo_kind":"official","path":"src/instruct_data_prep.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/instruct_data_prep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"63f870a98c11420f"}},{"code_sha256_prefix":"0d5485e114c8ba34","entry":"granite_3_1_prompt_input","repo":"ibm/nestful","repo_kind":"official","path":"src/instruct_data_prep.py","file_url":"https://github.com/ibm/nestful/blob/HEAD/src/instruct_data_prep.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d5485e114c8ba34"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}