{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-25886","title":"RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement","arxiv_id":"2607.25886","date":"2026-07-28","proceeding":null,"authors":["Fanqing Meng","Lingxiao Du","Qiguang Chen","Ziqi Zhao","Haocheng Lu","Mengkang Hu","Michael Qizhe Shieh"],"abstract":"Recursive self-improvement requires turning evidence of model failures into better models. Data-centric post-training research entails diagnosing capability gaps, designing and validating training-data strategies, and learning from checkpoint feedback. Can LLM agents automate this loop? Existing benchmarks entangle research decisions with optimization, serving, evaluation, and systems implementation, obscuring agents' research capability. We introduce RSIBench-Data, a controlled benchmark of LLM agents as data-centric researchers with a fixed post-training stack. Agents iteratively revise training-data strategies for a fixed target model; training and serving use Tinker-backed services, official evaluation runs through Harbor and E2B sandboxes, and budgets are fixed across agents. We evaluate four frontier agents on six benchmarks across software engineering, terminal use, scientific question answering, and mathematics. Agents demonstrate core data-centric research capabilities: in 58.33\\% of settings, they improve upon the first valid attempt by refining strategies from feedback. However, improvement is inconsistent. Among searches continuing after the best observed score, 78.26\\% end with a lower-scoring final attempt, while the rest only recover the same peak. A strong candidate may therefore appear early or midway through a run even as later revisions fail. Trajectory analysis identifies four patterns in stronger runs: accurate hypotheses, validation-grounded supervision, behavior-aligned data, and preservation of strong checkpoints. These findings suggest that current agents can make useful data-centric discoveries but cannot yet translate feedback into consistent improvements. RSIBench-Data provides a measurable, auditable testbed for the research capabilities required for recursive self-improvement. We open-source our code at https://github.com/evolvent-ai/RSIBench-Data.","url_abs":"https://arxiv.org/abs/2607.25886","url_pdf":"https://arxiv.org/pdf/2607.25886","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2607.25886","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.25886"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/evolvent-ai/RSIBench-Data","reach":null}],"summary":{"ran":23,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":25,"ran":23,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":25,"samples":[{"code_sha256_prefix":"9bab286b3b92cbc8","entry":"anthropic_messages_url","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/claude_usage_client.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/claude_usage_client.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9bab286b3b92cbc8"}},{"code_sha256_prefix":"c2b6300927eba11d","entry":"attempt_status","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/update_budget_state.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/update_budget_state.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2b6300927eba11d"}},{"code_sha256_prefix":"2f9a4291286c6671","entry":"estimate_tinker_cost","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/update_budget_state.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/update_budget_state.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f9a4291286c6671"}},{"code_sha256_prefix":"6520ef387a71df87","entry":"file_metadata","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/audit_agent_tamper.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/audit_agent_tamper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6520ef387a71df87"}},{"code_sha256_prefix":"ff474a9db6f4b246","entry":"is_in_flight","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/print_budget_status.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/print_budget_status.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff474a9db6f4b246"}},{"code_sha256_prefix":"7343f2d6438263f3","entry":"load_and_validate","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/validate_agent_run_config.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/validate_agent_run_config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7343f2d6438263f3"}},{"code_sha256_prefix":"0e02ec0c17b4922e","entry":"load_benchmark_spec","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/benchmark_spec.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/benchmark_spec.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e02ec0c17b4922e"}},{"code_sha256_prefix":"d5dd7ecfdc03e5b3","entry":"model_list_response","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/codex_responses_proxy.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/codex_responses_proxy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d5dd7ecfdc03e5b3"}},{"code_sha256_prefix":"dcca4150b043a010","entry":"normalize_legacy_keys","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/validate_agent_run_config.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/validate_agent_run_config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dcca4150b043a010"}},{"code_sha256_prefix":"cd4bcd1075062c4a","entry":"normalize_spec","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/benchmark_spec.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/benchmark_spec.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd4bcd1075062c4a"}},{"code_sha256_prefix":"796b0fb8c704c558","entry":"parse_float","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/validate_agent_run_config.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/validate_agent_run_config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"796b0fb8c704c558"}},{"code_sha256_prefix":"7e4d2b90c426e635","entry":"read_harbor_score","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/update_budget_state.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/update_budget_state.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e4d2b90c426e635"}},{"code_sha256_prefix":"60fefa17bbe708f9","entry":"read_json","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/validate_final_selection.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/validate_final_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60fefa17bbe708f9"}},{"code_sha256_prefix":"faa580358c930494","entry":"responses_url","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/codex_responses_proxy.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/codex_responses_proxy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"faa580358c930494"}},{"code_sha256_prefix":"e697492d6365185a","entry":"session_dir_for_budget_state","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/print_budget_status.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/print_budget_status.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e697492d6365185a"}},{"code_sha256_prefix":"cb1f4090fbd1ecb5","entry":"sha256_file","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/audit_agent_tamper.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/audit_agent_tamper.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb1f4090fbd1ecb5"}},{"code_sha256_prefix":"0bf2b1c4ef09876d","entry":"should_skip_file","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/audit_agent_tamper.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/audit_agent_tamper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bf2b1c4ef09876d"}},{"code_sha256_prefix":"5a2f021508d123d1","entry":"string_value","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/benchmark_spec.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/benchmark_spec.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a2f021508d123d1"}},{"code_sha256_prefix":"013dfba0b69672ce","entry":"summarize_claude_code","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/summarize_data_agent_usage.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/summarize_data_agent_usage.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"013dfba0b69672ce"}},{"code_sha256_prefix":"f92d0c6643759927","entry":"summarize_codex","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/summarize_data_agent_usage.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/summarize_data_agent_usage.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f92d0c6643759927"}},{"code_sha256_prefix":"f8b281f629fbfa53","entry":"summarize_datagen_script_usage","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/summarize_data_agent_usage.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/summarize_data_agent_usage.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8b281f629fbfa53"}},{"code_sha256_prefix":"c3bc72e5c22f5ee5","entry":"usage_counts","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/claude_usage_client.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/claude_usage_client.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3bc72e5c22f5ee5"}},{"code_sha256_prefix":"ddfc940006dcefa3","entry":"validate_messages_jsonl","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/validate_tinker_messages.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/validate_tinker_messages.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddfc940006dcefa3"}},{"code_sha256_prefix":"04abfa52ef92cfca","entry":"anthropic_messages_create","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/claude_usage_client.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/claude_usage_client.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04abfa52ef92cfca"}},{"code_sha256_prefix":"af7776dfd5634731","entry":"validate_attempt_selection","repo":"evolvent-ai/RSIBench-Data","repo_kind":"found_in_text","path":"runner/lib/validate_final_selection.py","file_url":"https://github.com/evolvent-ai/RSIBench-Data/blob/HEAD/runner/lib/validate_final_selection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af7776dfd5634731"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":1,"rejected_by_independent_check":0,"refused_by_a_rule":1,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}