{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stabletoolbench-towards-stable-large-scale","title":"StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models","arxiv_id":"2403.07714","date":"2024-03-12","proceeding":null,"authors":["Zhicheng Guo","Sijie Cheng","Hao Wang","Shihao Liang","Yujia Qin","Peng Li","Zhiyuan Liu","Maosong Sun","Yang Liu"],"abstract":"Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.","url_abs":"https://arxiv.org/abs/2403.07714v4","url_pdf":"https://arxiv.org/pdf/2403.07714v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stabletoolbench-towards-stable-large-scale","repo_url":"https://github.com/thunlp-mt/stabletoolbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"stabletoolbench-towards-stable-large-scale","repo_url":"https://github.com/zhichengg/stabletoolbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"stabletoolbench-towards-stable-large-scale","repo_url":"https://github.com/openbmb/toolbench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"stabletoolbench-towards-stable-large-scale","repo_url":"https://github.com/pwc-1/Paper-10/tree/main/stablelm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[{"slug":"stabletoolbench","name":"StableToolBench","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.07714","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.07714"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pwc-1/Paper-10/tree/main/stablelm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp-mt/stabletoolbench","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhichengg/stabletoolbench","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openbmb/toolbench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"ran":5,"ran_violates":1,"unverified":7},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":12,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7b6b723a33f5ea2a","entry":"change_name","repo":"thunlp-mt/stabletoolbench","repo_kind":"official","path":"server/utils.py","file_url":"https://github.com/thunlp-mt/stabletoolbench/blob/HEAD/server/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b6b723a33f5ea2a"}},{"code_sha256_prefix":"b57a77da7c53ac7a","entry":"chat_completion_request","repo":"openbmb/toolbench","repo_kind":"listed","path":"toolbench/inference/LLM/chatgpt_function_model.py","file_url":"https://github.com/openbmb/toolbench/blob/HEAD/toolbench/inference/LLM/chatgpt_function_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b57a77da7c53ac7a"}},{"code_sha256_prefix":"709a780a94ca2670","entry":"get_compressed_list","repo":"openbmb/toolbench","repo_kind":"listed","path":"toolbench/model/compression.py","file_url":"https://github.com/openbmb/toolbench/blob/HEAD/toolbench/model/compression.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"709a780a94ca2670"}},{"code_sha256_prefix":"be8785d230bc7581","entry":"is_valid_json","repo":"thunlp-mt/stabletoolbench","repo_kind":"official","path":"server/main_mirrorapi.py","file_url":"https://github.com/thunlp-mt/stabletoolbench/blob/HEAD/server/main_mirrorapi.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be8785d230bc7581"}},{"code_sha256_prefix":"a73f07e24a8b8a85","entry":"process_system_message","repo":"openbmb/toolbench","repo_kind":"listed","path":"toolbench/utils.py","file_url":"https://github.com/openbmb/toolbench/blob/HEAD/toolbench/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a73f07e24a8b8a85"}},{"code_sha256_prefix":"0bf414a2b44babe6","entry":"raise_warning_for_incompatible_cpu_offloading_configuration","repo":"openbmb/toolbench","repo_kind":"listed","path":"toolbench/model/model_adapter.py","file_url":"https://github.com/openbmb/toolbench/blob/HEAD/toolbench/model/model_adapter.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0bf414a2b44babe6"}},{"code_sha256_prefix":"e6d1b09cb3624d3f","entry":"standardize","repo":"thunlp-mt/stabletoolbench","repo_kind":"official","path":"server/utils.py","file_url":"https://github.com/thunlp-mt/stabletoolbench/blob/HEAD/server/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e6d1b09cb3624d3f"}},{"code_sha256_prefix":"04eed77ef8a3068b","entry":"standardize_category","repo":"openbmb/toolbench","repo_kind":"listed","path":"toolbench/utils.py","file_url":"https://github.com/openbmb/toolbench/blob/HEAD/toolbench/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"04eed77ef8a3068b"}},{"code_sha256_prefix":"2901dfbf4ce068d8","entry":"c2p_dynamic_expand","repo":"pwc-1/Paper-10","repo_kind":"listed","path":"sew_d/modeling_sew_d.py","file_url":"https://github.com/pwc-1/Paper-10/blob/HEAD/sew_d/modeling_sew_d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2901dfbf4ce068d8"}},{"code_sha256_prefix":"386536152177a379","entry":"compress","repo":"openbmb/toolbench","repo_kind":"listed","path":"toolbench/model/compression.py","file_url":"https://github.com/openbmb/toolbench/blob/HEAD/toolbench/model/compression.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"386536152177a379"}},{"code_sha256_prefix":"a110d4f6e1706068","entry":"format_speech_generation_kwargs","repo":"pwc-1/Paper-10","repo_kind":"listed","path":"seamless_m4t/modeling_seamless_m4t.py","file_url":"https://github.com/pwc-1/Paper-10/blob/HEAD/seamless_m4t/modeling_seamless_m4t.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a110d4f6e1706068"}},{"code_sha256_prefix":"8b5f9e0a6ad43062","entry":"format_speech_generation_kwargs","repo":"pwc-1/Paper-10","repo_kind":"listed","path":"seamless_m4t_v2/modeling_seamless_m4t_v2.py","file_url":"https://github.com/pwc-1/Paper-10/blob/HEAD/seamless_m4t_v2/modeling_seamless_m4t_v2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b5f9e0a6ad43062"}},{"code_sha256_prefix":"eab40e1435b85709","entry":"get_conv_template","repo":"openbmb/toolbench","repo_kind":"listed","path":"toolbench/tool_conversation.py","file_url":"https://github.com/openbmb/toolbench/blob/HEAD/toolbench/tool_conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eab40e1435b85709"}},{"code_sha256_prefix":"2f3cb5d0dcc18502","entry":"get_gpu_memory","repo":"openbmb/toolbench","repo_kind":"listed","path":"toolbench/utils.py","file_url":"https://github.com/openbmb/toolbench/blob/HEAD/toolbench/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f3cb5d0dcc18502"}},{"code_sha256_prefix":"5f9d4d17fb70af2b","entry":"load_compress_model","repo":"openbmb/toolbench","repo_kind":"listed","path":"toolbench/model/compression.py","file_url":"https://github.com/openbmb/toolbench/blob/HEAD/toolbench/model/compression.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f9d4d17fb70af2b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}