{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rotbench-a-multi-level-benchmark-for","title":"RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning","arxiv_id":"2401.08326","date":"2024-01-16","proceeding":null,"authors":["Junjie Ye","Yilong Wu","Songyang Gao","Caishuang Huang","Sixian Li","Guanyu Li","Xiaoran Fan","Qi Zhang","Tao Gui","Xuanjing Huang"],"abstract":"Tool learning has generated widespread interest as a vital means of interaction between Large Language Models (LLMs) and the physical world. Current research predominantly emphasizes LLMs' capacity to utilize tools in well-structured environments while overlooking their stability when confronted with the inevitable noise of the real world. To bridge this gap, we introduce RoTBench, a multi-level benchmark for evaluating the robustness of LLMs in tool learning. Specifically, we establish five external environments, each featuring varying levels of noise (i.e., Clean, Slight, Medium, Heavy, and Union), providing an in-depth analysis of the model's resilience across three critical phases: tool selection, parameter identification, and content filling. Experiments involving six widely-used models underscore the urgent necessity for enhancing the robustness of LLMs in tool learning. For instance, the performance of GPT-4 even drops significantly from 80.00 to 58.10 when there is no substantial change in manual accuracy. More surprisingly, the noise correction capability inherent in the GPT family paradoxically impedes its adaptability in the face of mild noise. In light of these findings, we propose RoTTuning, a strategy that enriches the diversity of training environments to bolster the robustness of LLMs in tool learning. The code and data are available at https://github.com/Junjie-Ye/RoTBench.","url_abs":"https://arxiv.org/abs/2401.08326v3","url_pdf":"https://arxiv.org/pdf/2401.08326v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rotbench-a-multi-level-benchmark-for","repo_url":"https://github.com/junjie-ye/rotbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.08326","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.08326"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Junjie-Ye/RoTBench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"ran_violates":1,"unverified":5},"by_repo_kind":{"official":{"samples":15,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b8f6a0e07e142f30","entry":"compress_prompt","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/Inference/models.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/Inference/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8f6a0e07e142f30"}},{"code_sha256_prefix":"c6316fc799a85487","entry":"conversations_process","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/Data_Process/transform_process.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/Data_Process/transform_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c6316fc799a85487"}},{"code_sha256_prefix":"f6309633583a2931","entry":"get_answer_list","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/Evaluation/evaluate.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/Evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6309633583a2931"}},{"code_sha256_prefix":"31049a1bb3c726a8","entry":"get_cata_list","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/Evaluation/evaluate.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/Evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"31049a1bb3c726a8"}},{"code_sha256_prefix":"709a780a94ca2670","entry":"get_compressed_list","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/model/compression.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/model/compression.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"709a780a94ca2670"}},{"code_sha256_prefix":"b7e2a5655a7335a3","entry":"mode_control","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/Data_Process/transform_process.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/Data_Process/transform_process.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7e2a5655a7335a3"}},{"code_sha256_prefix":"0bf414a2b44babe6","entry":"raise_warning_for_incompatible_cpu_offloading_configuration","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/model/model_adapter.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/model/model_adapter.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0bf414a2b44babe6"}},{"code_sha256_prefix":"b5daeb59f8c9cfd0","entry":"random_name","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/Data_Process/transformer.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/Data_Process/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5daeb59f8c9cfd0"}},{"code_sha256_prefix":"78e64427cacb1728","entry":"reverse","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/Data_Process/transformer.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/Data_Process/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"78e64427cacb1728"}},{"code_sha256_prefix":"9f30e5c30d04940e","entry":"tool_slight_transform","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/Data_Process/transformer.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/Data_Process/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f30e5c30d04940e"}},{"code_sha256_prefix":"386536152177a379","entry":"compress","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/model/compression.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/model/compression.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"386536152177a379"}},{"code_sha256_prefix":"bbfc0fbb6c40d003","entry":"get_config","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/Evaluation/evaluate.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/Evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbfc0fbb6c40d003"}},{"code_sha256_prefix":"6aa619d359276223","entry":"get_conv_template","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/tool_conversation.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/tool_conversation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6aa619d359276223"}},{"code_sha256_prefix":"2f3cb5d0dcc18502","entry":"get_gpu_memory","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/utils.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f3cb5d0dcc18502"}},{"code_sha256_prefix":"5f9d4d17fb70af2b","entry":"load_compress_model","repo":"Junjie-Ye/RoTBench","repo_kind":"official","path":"Code/model/compression.py","file_url":"https://github.com/Junjie-Ye/RoTBench/blob/HEAD/Code/model/compression.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f9d4d17fb70af2b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}