{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tapilot-crossing-benchmarking-and-evolving","title":"Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents","arxiv_id":"2403.05307","date":"2024-03-08","proceeding":null,"authors":["Jinyang Li","Nan Huo","Yan Gao","Jiayi Shi","Yingxiu Zhao","Ge Qu","Yurong Wu","Chenhao Ma","Jian-Guang Lou","Reynold Cheng"],"abstract":"Interactive Data Analysis, the collaboration between humans and LLM agents, enables real-time data exploration for informed decision-making. The challenges and costs of collecting realistic interactive logs for data analysis hinder the quantitative evaluation of Large Language Model (LLM) agents in this task. To mitigate this issue, we introduce Tapilot-Crossing, a new benchmark to evaluate LLM agents on interactive data analysis. Tapilot-Crossing contains 1024 interactions, covering 4 practical scenarios: Normal, Action, Private, and Private Action. Notably, Tapilot-Crossing is constructed by an economical multi-agent environment, Decision Company, with few human efforts. We evaluate popular and advanced LLM agents in Tapilot-Crossing, which underscores the challenges of interactive data analysis. Furthermore, we propose Adaptive Interaction Reflection (AIR), a self-generated reflection strategy that guides LLM agents to learn from successful history. Experiments demonstrate that Air can evolve LLMs into effective interactive data analysis agents, achieving a relative performance improvement of up to 44.5%.","url_abs":"https://arxiv.org/abs/2403.05307v1","url_pdf":"https://arxiv.org/pdf/2403.05307v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tapilot-crossing-benchmarking-and-evolving","repo_url":"https://github.com/tapilot-crossing/tapilot_code","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.05307","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.05307"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tapilot-crossing/tapilot_code","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":16},"by_repo_kind":{"official":{"samples":16,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8c5a9f85651f57df","entry":"extract_answer","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"eval/eval_multi_choice.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/eval/eval_multi_choice.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c5a9f85651f57df"}},{"code_sha256_prefix":"c805463b7efcc6de","entry":"extract_code","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"postprocessing/combine_code_gen_AIR.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/postprocessing/combine_code_gen_AIR.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c805463b7efcc6de"}},{"code_sha256_prefix":"0d2c5937f51ac5ab","entry":"extract_code","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"postprocessing/combine_code_gen_base.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/postprocessing/combine_code_gen_base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d2c5937f51ac5ab"}},{"code_sha256_prefix":"7289a9d629da016d","entry":"extract_python_code","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"postprocessing/combine_code_gen_COT.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/postprocessing/combine_code_gen_COT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7289a9d629da016d"}},{"code_sha256_prefix":"4b184cec1d20b4b1","entry":"format_code","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"postprocessing/combine_code_gen_AIR.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/postprocessing/combine_code_gen_AIR.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b184cec1d20b4b1"}},{"code_sha256_prefix":"3373c6a0b725a712","entry":"get_llm_response","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"methods/utils.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/methods/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3373c6a0b725a712"}},{"code_sha256_prefix":"8fc4a6d9995f1727","entry":"is_float","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"eval/eval_code_gen.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/eval/eval_code_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fc4a6d9995f1727"}},{"code_sha256_prefix":"af35b7d82ed3408a","entry":"is_folder_empty","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"eval/eval_multi_choice.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/eval/eval_multi_choice.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af35b7d82ed3408a"}},{"code_sha256_prefix":"17f3b85b87f477d7","entry":"is_penultimate_directory","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"eval/eval_code_gen.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/eval/eval_code_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17f3b85b87f477d7"}},{"code_sha256_prefix":"5966028efcba3e7a","entry":"parse_chain","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"eval/eval_code_gen.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/eval/eval_code_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5966028efcba3e7a"}},{"code_sha256_prefix":"2143bcd79a8030f5","entry":"parse_reference_answer_content","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"eval/eval_multi_choice.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/eval/eval_multi_choice.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2143bcd79a8030f5"}},{"code_sha256_prefix":"a52f99f2c2698f9d","entry":"prepare_code_head","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"postprocessing/combine_code_gen_COT.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/postprocessing/combine_code_gen_COT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a52f99f2c2698f9d"}},{"code_sha256_prefix":"bde1ccb6e2bc9262","entry":"prepare_code_head","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"postprocessing/combine_code_gen_base.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/postprocessing/combine_code_gen_base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bde1ccb6e2bc9262"}},{"code_sha256_prefix":"9ebe5ad9b6ac2548","entry":"prepare_private_funcs","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"postprocessing/combine_code_gen_base.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/postprocessing/combine_code_gen_base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ebe5ad9b6ac2548"}},{"code_sha256_prefix":"1e9bc7efa36c49db","entry":"process_python_output","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"methods/utils.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/methods/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e9bc7efa36c49db"}},{"code_sha256_prefix":"12ca59656e73c371","entry":"remove_read_csv","repo":"tapilot-crossing/tapilot_code","repo_kind":"official","path":"postprocessing/combine_code_gen_COT.py","file_url":"https://github.com/tapilot-crossing/tapilot_code/blob/HEAD/postprocessing/combine_code_gen_COT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12ca59656e73c371"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}