{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/intentest-stress-testing-for-intent-integrity","title":"IntenTest: Stress Testing for Intent Integrity in API-Calling LLM Agents","arxiv_id":"2506.07524","date":"2025-06-09","proceeding":null,"authors":["Shiwei Feng","Xiangzhe Xu","Xuan Chen","Kaiyuan Zhang","Syed Yusuf Ahmed","Zian Su","Mingwei Zheng","Xiangyu Zhang"],"abstract":"LLM agents are increasingly deployed to automate real-world tasks by invoking APIs through natural language instructions. While powerful, they often suffer from misinterpretation of user intent, leading to the agent's actions that diverge from the user's intended goal, especially as external toolkits evolve. Traditional software testing assumes structured inputs and thus falls short in handling the ambiguity of natural language. We introduce IntenTest, an API-centric stress testing framework that systematically uncovers intent integrity violations in LLM agents. Unlike prior work focused on fixed benchmarks or adversarial inputs, IntenTest generates realistic tasks based on toolkits' documentation and applies targeted mutations to expose subtle agent errors while preserving user intent. To guide testing, we propose semantic partitioning, which organizes natural language tasks into meaningful categories based on toolkit API parameters and their equivalence classes. Within each partition, seed tasks are mutated and ranked by a lightweight predictor that estimates the likelihood of triggering agent errors. To enhance efficiency, IntenTest maintains a datatype-aware strategy memory that retrieves and adapts effective mutation patterns from past cases. Experiments on 80 toolkit APIs demonstrate that IntenTest effectively uncovers intent integrity violations, significantly outperforming baselines in both error-exposing rate and query efficiency. Moreover, IntenTest generalizes well to stronger target models using smaller LLMs for test generation, and adapts to evolving APIs across domains.","url_abs":"https://arxiv.org/abs/2506.07524v1","url_pdf":"https://arxiv.org/pdf/2506.07524v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"software-testing","task_name":"software testing"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.07524","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.07524"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/OthersideAI/self-operating-computer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"found_in_text":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8fb25821e93bbc5a","entry":"convert_percent_to_decimal","repo":"OthersideAI/self-operating-computer","repo_kind":"found_in_text","path":"operate/utils/misc.py","file_url":"https://github.com/OthersideAI/self-operating-computer/blob/HEAD/operate/utils/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fb25821e93bbc5a"}},{"code_sha256_prefix":"a50e3b454b555b94","entry":"evaluate_final_screenshot","repo":"OthersideAI/self-operating-computer","repo_kind":"found_in_text","path":"evaluate.py","file_url":"https://github.com/OthersideAI/self-operating-computer/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a50e3b454b555b94"}},{"code_sha256_prefix":"d85dd878596925b2","entry":"format_evaluation_prompt","repo":"OthersideAI/self-operating-computer","repo_kind":"found_in_text","path":"evaluate.py","file_url":"https://github.com/OthersideAI/self-operating-computer/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d85dd878596925b2"}},{"code_sha256_prefix":"06053a203aa2ddc3","entry":"get_label_coordinates","repo":"OthersideAI/self-operating-computer","repo_kind":"found_in_text","path":"operate/utils/label.py","file_url":"https://github.com/OthersideAI/self-operating-computer/blob/HEAD/operate/utils/label.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06053a203aa2ddc3"}},{"code_sha256_prefix":"95bfb8d8ec9060ff","entry":"get_text_coordinates","repo":"OthersideAI/self-operating-computer","repo_kind":"found_in_text","path":"operate/utils/ocr.py","file_url":"https://github.com/OthersideAI/self-operating-computer/blob/HEAD/operate/utils/ocr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95bfb8d8ec9060ff"}},{"code_sha256_prefix":"3ddd1a66cc3ae401","entry":"is_overlapping","repo":"OthersideAI/self-operating-computer","repo_kind":"found_in_text","path":"operate/utils/label.py","file_url":"https://github.com/OthersideAI/self-operating-computer/blob/HEAD/operate/utils/label.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ddd1a66cc3ae401"}},{"code_sha256_prefix":"9bf6ad9eec22b696","entry":"parse_eval_content","repo":"OthersideAI/self-operating-computer","repo_kind":"found_in_text","path":"evaluate.py","file_url":"https://github.com/OthersideAI/self-operating-computer/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9bf6ad9eec22b696"}},{"code_sha256_prefix":"b91088edc90ff80d","entry":"parse_operations","repo":"OthersideAI/self-operating-computer","repo_kind":"found_in_text","path":"operate/utils/misc.py","file_url":"https://github.com/OthersideAI/self-operating-computer/blob/HEAD/operate/utils/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b91088edc90ff80d"}},{"code_sha256_prefix":"1a44feb7665fac56","entry":"validate_and_extract_image_data","repo":"OthersideAI/self-operating-computer","repo_kind":"found_in_text","path":"operate/utils/label.py","file_url":"https://github.com/OthersideAI/self-operating-computer/blob/HEAD/operate/utils/label.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a44feb7665fac56"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}