{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-14600","title":"SciPaths: Forecasting Pathways to Scientific Discovery","arxiv_id":"2605.14600","date":"2026-05-14","proceeding":null,"authors":["Eric Chamoun","Yizhou Chi","Yulong Chen","Rui Cao","Zifeng Ding","Michalis Korakakis","Andreas Vlachos"],"abstract":"Scientific progress depends on sequences of enabling contributions, yet existing AI4Science benchmarks largely focus on citation prediction, literature retrieval, or idea generation rather than the dependencies that make progress possible. In this paper, we introduce discovery pathway forecasting: given a target scientific contribution and the prior literature available at a specified time, the task is to (1) identify the enabling contributions required to realize it and (2) ground each in prior work when such prior work exists. We present SciPaths, a benchmark of 262 expert-annotated gold pathways and 2,444 silver pathways constructed from machine learning and natural language processing papers, where each pathway records enabling contributions, roles, rationales, and prior-work groundings or unmapped decisions. Evaluating frontier and open-weight language models, we find that the best model reaches only 0.189 F1 under strict semantic matching, with core methodological dependencies hardest to recover. Prior-work grounding improves substantially when gold enabling contributions are provided, showing that decomposition quality is a major bottleneck for end-to-end pathway recovery. SciPaths therefore shifts evaluation toward a missing capability in scientific forecasting: reasoning backward from a target contribution to the enabling scientific building blocks and prior-work dependencies that make it feasible.","url_abs":"https://arxiv.org/abs/2605.14600","url_pdf":"https://arxiv.org/pdf/2605.14600","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.14600","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.14600"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ericchamoun/scipaths","reach":null}],"summary":{"ran_draft_wrong":7,"ran_fixture":1,"ran_honours":1,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6b0764930fb7b43b","entry":"_adapter_id","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b0764930fb7b43b"}},{"code_sha256_prefix":"41dd7c80809a6b4b","entry":"_clean_ingredients","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41dd7c80809a6b4b"}},{"code_sha256_prefix":"fdae5322ce7c33a4","entry":"_format_few_shot_examples","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdae5322ce7c33a4"}},{"code_sha256_prefix":"f9ab17aaea26d223","entry":"_format_references","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9ab17aaea26d223"}},{"code_sha256_prefix":"81edaf7e6a422800","entry":"_load_model","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81edaf7e6a422800"}},{"code_sha256_prefix":"d9712c30c035b5fd","entry":"_max_context_length","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9712c30c035b5fd"}},{"code_sha256_prefix":"3067e769456a3f17","entry":"_model_id","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3067e769456a3f17"}},{"code_sha256_prefix":"d336c3676bdf294f","entry":"_parse_ingredients","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d336c3676bdf294f"}},{"code_sha256_prefix":"3b3afe0ea8937b31","entry":"_repair_ingredients","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b3afe0ea8937b31"}},{"code_sha256_prefix":"a90378fb8222f5e4","entry":"_build_prompt","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a90378fb8222f5e4"}},{"code_sha256_prefix":"39b9f0e377cc8f4b","entry":"generate_ingredients","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39b9f0e377cc8f4b"}},{"code_sha256_prefix":"76b9563e4232e71f","entry":"generate_local_hf","repo":"ericchamoun/scipaths","repo_kind":"found_in_text","path":"benchmark_eval/modules/generator.py","file_url":"https://github.com/ericchamoun/scipaths/blob/HEAD/benchmark_eval/modules/generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76b9563e4232e71f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}