{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-15766","title":"Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery","arxiv_id":"2607.15766","date":"2026-07-17","proceeding":null,"authors":["Tianyun Zhong","Wangyi Jiang","Wei Wang","Xuanang Chen","Yaojie Lu","Shiwei Ye","Yuzhen Shi","Boyu Yang","Jinghang Wang","Han Li","Weiqi Zhai","Bing Zhao","Hu Wei","Haiyang Yu","Yongbin Li","Hongyu Lin","Le Sun","Xianpei Han"],"abstract":"Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured. We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discriminative, and testable hypothesis spaces from inconclusive evidence, including anomalous observations and fragmented records, to guide subsequent investigation. To evaluate this capability, we introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets. To construct HypoData at scale, we propose Retrospective Context Regression, a Forge--Audit pipeline that reconstructs pre-conclusion contexts from completed expert documents by removing explicit conclusions, target hypotheses, and retrospective causal attributions while preserving the factual substrate. Because PHD admits multiple valid outputs, HypoEval combines bidirectional pairwise judgments with Bradley--Terry--Davidson aggregation for ranking and six-dimensional rubric scoring for diagnosis. Experiments on 15 frontier LLMs reveal clear capability stratification and model-dependent effects of structured analytical skills, with gains for several lower-performing models on HypoArena but regressions for other systems, including a top-performing model. Compared with absolute rubric scoring, arena evaluation resolves finer-grained differences among models, with aggregated rankings showing strong agreement with human experts and an independent judge. Together, these results support treating PHD as a distinct target for evaluating how LLMs formulate investigative directions when final conclusions are withheld. Our code and data are publicly available at github.com/SKYLENAGE-AI/HypoArena and github.com/SKYLENAGE-AI/HypoArena.","url_abs":"https://arxiv.org/abs/2607.15766","url_pdf":"https://arxiv.org/pdf/2607.15766","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2607.15766","atlas_url":"https://app.syntology.ai/?focus=2607.15766","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.15766"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/SKYLENAGE-AI/HypoArena","reach":null}],"summary":{"ran":6,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"0f4594bfb6ac6fc2","entry":"cases_path","repo":"SKYLENAGE-AI/HypoArena","repo_kind":"found_in_text","path":"basics/paths.py","file_url":"https://github.com/SKYLENAGE-AI/HypoArena/blob/HEAD/basics/paths.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f4594bfb6ac6fc2"}},{"code_sha256_prefix":"13c24c42e8e6e336","entry":"coerce_to_model","repo":"SKYLENAGE-AI/HypoArena","repo_kind":"found_in_text","path":"basics/parsing.py","file_url":"https://github.com/SKYLENAGE-AI/HypoArena/blob/HEAD/basics/parsing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"13c24c42e8e6e336"}},{"code_sha256_prefix":"0e0a2bc6bba702c9","entry":"existing_ids","repo":"SKYLENAGE-AI/HypoArena","repo_kind":"found_in_text","path":"basics/io.py","file_url":"https://github.com/SKYLENAGE-AI/HypoArena/blob/HEAD/basics/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e0a2bc6bba702c9"}},{"code_sha256_prefix":"2f41dc825738de71","entry":"extract_json","repo":"SKYLENAGE-AI/HypoArena","repo_kind":"found_in_text","path":"basics/parsing.py","file_url":"https://github.com/SKYLENAGE-AI/HypoArena/blob/HEAD/basics/parsing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f41dc825738de71"}},{"code_sha256_prefix":"56d4e266de2a43d4","entry":"keep_in_pool","repo":"SKYLENAGE-AI/HypoArena","repo_kind":"found_in_text","path":"basics/schema.py","file_url":"https://github.com/SKYLENAGE-AI/HypoArena/blob/HEAD/basics/schema.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56d4e266de2a43d4"}},{"code_sha256_prefix":"8bbcd90ecfd416a7","entry":"source_dir","repo":"SKYLENAGE-AI/HypoArena","repo_kind":"found_in_text","path":"basics/paths.py","file_url":"https://github.com/SKYLENAGE-AI/HypoArena/blob/HEAD/basics/paths.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8bbcd90ecfd416a7"}},{"code_sha256_prefix":"a2162894ffed007a","entry":"get_domain","repo":"SKYLENAGE-AI/HypoArena","repo_kind":"found_in_text","path":"basics/domain.py","file_url":"https://github.com/SKYLENAGE-AI/HypoArena/blob/HEAD/basics/domain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2162894ffed007a"}},{"code_sha256_prefix":"ba9c71c1b76f831d","entry":"is_available","repo":"SKYLENAGE-AI/HypoArena","repo_kind":"found_in_text","path":"basics/platform.py","file_url":"https://github.com/SKYLENAGE-AI/HypoArena/blob/HEAD/basics/platform.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba9c71c1b76f831d"}},{"code_sha256_prefix":"ee946ffb793b6b7d","entry":"source_metadata_path","repo":"SKYLENAGE-AI/HypoArena","repo_kind":"found_in_text","path":"basics/io.py","file_url":"https://github.com/SKYLENAGE-AI/HypoArena/blob/HEAD/basics/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee946ffb793b6b7d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":1,"rejected_by_independent_check":0,"refused_by_a_rule":1,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}