{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-08557","title":"OpenVisTool: An Open Recipe for Synthesizing Instructive Visual Tool-Use Trajectories","arxiv_id":"2608.08557","date":"2026-08-09","proceeding":null,"authors":["Changhao Xiang","Shilin Zhang","Zheng Ma","Kanzhi Cheng","Ruize Ma","Yi Feng","Jianbing Zhang","Zhi Wang","Zhen Wu","Xinyu Dai","Lewei Lu"],"abstract":"Visual tool use has emerged as a fundamental capability for multimodal agents to actively acquire evidence beyond a fixed image encoding. The prevailing recipe learns this capability from teacher-generated trajectories filtered for answer correctness, implicitly assuming that every successful demonstration provides effective supervision. We argue this assumption is flawed: a strong teacher often reaches the correct answer without needing its tool calls, and imitating such trajectories teaches a student that tool calls accompany correct answers, not that tool observations ground them. We present OpenVisTool, an open framework for constructing instructive visual tool-use trajectories that provide effective supervision for tool learning. The key insight is that a trajectory should be retained only if its answer is correct (outcome validity) and its tool observations causally contribute to that answer (causal utility). The framework operates in three stages: difficulty screening to select queries that are not reliably answerable without tools, domain-specific trajectory synthesis to elicit coherent tool-use trajectories, and supervision verification to jointly test both conditions. Rather than encouraging models to imitate tool calls, the resulting supervision teaches when and how visual evidence should be acquired. Using this framework, we construct OpenVisTool-42K, a dataset spanning five visual reasoning domains, together with OpenVisTool-Bench, a benchmark covering the same domains. Across four backbones (4B-27B), fine-tuning on OpenVisTool-42K consistently improves visual tool-use performance and yields gains on two out-of-distribution benchmarks; the larger models approach leading closed-source systems. The evidence suggests that effective visual tool use is learned from causally grounded supervision rather than tool-calling patterns.","url_abs":"https://arxiv.org/abs/2608.08557","url_pdf":"https://arxiv.org/pdf/2608.08557","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2608.08557","atlas_url":"https://app.syntology.ai/?focus=2608.08557","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.08557"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Changhao-Xiang/OpenVisTool","reach":null}],"summary":{"ran":18,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":19,"ran":18,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"00bd2a4e062805e9","entry":"extract_bold","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/evaluate_answer_rule.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/evaluate_answer_rule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00bd2a4e062805e9"}},{"code_sha256_prefix":"f1e5290562997ded","entry":"extract_boxed","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/evaluate_answer_rule.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/evaluate_answer_rule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1e5290562997ded"}},{"code_sha256_prefix":"c5bb07f1ea986eee","entry":"extract_events","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/filter_acc_toolcall.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/filter_acc_toolcall.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5bb07f1ea986eee"}},{"code_sha256_prefix":"85cec999c106b6cb","entry":"extract_media_dir","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/filter_acc_toolcall.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/filter_acc_toolcall.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85cec999c106b6cb"}},{"code_sha256_prefix":"6038c566486bee60","entry":"get_query_value","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/dataset.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6038c566486bee60"}},{"code_sha256_prefix":"916a9436f2b9ec5b","entry":"get_session_key","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/dataset.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"916a9436f2b9ec5b"}},{"code_sha256_prefix":"797ab67af1e2629e","entry":"get_text_field","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/filter_tool_gain_with_prefix.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/filter_tool_gain_with_prefix.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"797ab67af1e2629e"}},{"code_sha256_prefix":"2fafd3206765fbde","entry":"infer_media_dir","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/filter_difficulty.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/filter_difficulty.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fafd3206765fbde"}},{"code_sha256_prefix":"62a8ef9abafa8b5d","entry":"infer_output_path","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/filter_difficulty.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/filter_difficulty.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62a8ef9abafa8b5d"}},{"code_sha256_prefix":"654ceef66c81c477","entry":"infer_progress_path","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/filter_difficulty.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/filter_difficulty.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"654ceef66c81c477"}},{"code_sha256_prefix":"a49686bee6ec64ab","entry":"item_key","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/evaluate_instructive_trajectory_ablation.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/evaluate_instructive_trajectory_ablation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a49686bee6ec64ab"}},{"code_sha256_prefix":"fd449ff795ee3154","entry":"load_jsonl","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/evaluate_instructive_trajectory_ablation.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/evaluate_instructive_trajectory_ablation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd449ff795ee3154"}},{"code_sha256_prefix":"fa52ba8cf457edcf","entry":"load_jsonl","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/filter_tool_gain_with_prefix.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/filter_tool_gain_with_prefix.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa52ba8cf457edcf"}},{"code_sha256_prefix":"9dd07a6f35029d86","entry":"load_jsonl_records","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/dataset.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9dd07a6f35029d86"}},{"code_sha256_prefix":"24e4999eff337610","entry":"normalize_answer","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/evaluate_answer_rule.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/evaluate_answer_rule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24e4999eff337610"}},{"code_sha256_prefix":"a3750be5aa23e5d6","entry":"parse_sample_range","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/filter_tool_gain_with_prefix.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/filter_tool_gain_with_prefix.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3750be5aa23e5d6"}},{"code_sha256_prefix":"dec463a3077aa301","entry":"parse_trace","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/filter_acc_toolcall.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/filter_acc_toolcall.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dec463a3077aa301"}},{"code_sha256_prefix":"abad37ecded87349","entry":"task_key","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/evaluate_instructive_trajectory_ablation.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/evaluate_instructive_trajectory_ablation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"abad37ecded87349"}},{"code_sha256_prefix":"da73a8efc063e183","entry":"rule_check_html","repo":"Changhao-Xiang/OpenVisTool","repo_kind":"found_in_text","path":"distill/filter/evaluate_html_vlm_judge.py","file_url":"https://github.com/Changhao-Xiang/OpenVisTool/blob/HEAD/distill/filter/evaluate_html_vlm_judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da73a8efc063e183"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}