{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2609-00787","title":"StudyBench: Can Self-Evolution Squeeze Textbooks for Olympiad Capability?","arxiv_id":"2609.00787","date":"2026-09-01","proceeding":null,"authors":["Yinghao Chen","Zixi Chen","Bingxiang He","Ziqing Qiao","Huan-ang Gao","Yinuo Xu","Yuxin Zuo","Zeyuan Liu","Yuhao Zhan","Chaojun Xiao"],"abstract":"Humans need to study only a handful of well-written textbooks to master a discipline and attempt its hardest problems. We argue that an ideal self-evolution method should share the same property, that is autonomously learning from raw training material for transferable problem-solving capability. However, we still lack a direct measurement for it. We introduce StudyBench, a controlled physics benchmark that directly measures how efficiently a self-evolution method converts training material into capability. We organise the test set into an Application Set, consisting of difficult textbook problems and evaluating absorption ability, and a Transfer Set, consisting of olympiad-level problems and evaluating transfer ability. Benchmarking representative self-evolution methods across three base models, we find that improvements on the Application Set rarely translate to the harder Transfer Set. A guidance ablation exposes a Guidance Gap: even the strongest method closes only a small fraction of what the same material unlocks when supplied as in-context guidance. Besides, every method hits a Compute Plateau, saturating well before exhausting its compute budget. The remaining gap is therefore a method problem rather than a data or compute problem. By offering a clean and controlled benchmark, StudyBench turns self-evolution progress from an open-ended pursuit into a measurable target for future research. Our code is released at https://github.com/thunlp/StudyBench.","url_abs":"https://arxiv.org/abs/2609.00787","url_pdf":"https://arxiv.org/pdf/2609.00787","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2609.00787","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2609.00787"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/thunlp/StudyBench","reach":{"status":"ok"}}],"summary":{"ran_violates":1,"unverified":25},"by_repo_kind":{"found_in_text":{"samples":26,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":26,"samples":[{"code_sha256_prefix":"20063c675d638a77","entry":"is_equiv","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"eval/math_equivalence.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/eval/math_equivalence.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20063c675d638a77"}},{"code_sha256_prefix":"dc60ae5d710b98ab","entry":"already_done","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"corpus_scripts/mineru_parse.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/corpus_scripts/mineru_parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc60ae5d710b98ab"}},{"code_sha256_prefix":"f9956479d2011bbc","entry":"already_done","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"corpus_scripts/ocr_fix.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/corpus_scripts/ocr_fix.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9956479d2011bbc"}},{"code_sha256_prefix":"9a60e5840c4ed6ae","entry":"eval_keys","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"corpus_scripts/redact_eval.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/corpus_scripts/redact_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a60e5840c4ed6ae"}},{"code_sha256_prefix":"b1626be64f2fe39b","entry":"find_mds","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"corpus_scripts/ocr_fix.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/corpus_scripts/ocr_fix.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1626be64f2fe39b"}},{"code_sha256_prefix":"8124374e4a4daedf","entry":"find_pdfs","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"corpus_scripts/mineru_parse.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/corpus_scripts/mineru_parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8124374e4a4daedf"}},{"code_sha256_prefix":"0d82f56bb021e211","entry":"guidance_results","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/build_kp_guidance.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/build_kp_guidance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d82f56bb021e211"}},{"code_sha256_prefix":"9b280e4dc2463ca5","entry":"kp_hash","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/canonicalize_knowledge_points.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/canonicalize_knowledge_points.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b280e4dc2463ca5"}},{"code_sha256_prefix":"6a6b33dbd8b05fd2","entry":"load_hf_items","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"eval/run_benchmark.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/eval/run_benchmark.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a6b33dbd8b05fd2"}},{"code_sha256_prefix":"5f349754fa444a29","entry":"make_judge_backend","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"eval/judge.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/eval/judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f349754fa444a29"}},{"code_sha256_prefix":"bcbc70ffbbcb2e68","entry":"match_inline_example","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/build_textbook_fragments.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/build_textbook_fragments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bcbc70ffbbcb2e68"}},{"code_sha256_prefix":"4253ee8dab999331","entry":"norm_token","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"corpus_scripts/redact_eval.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/corpus_scripts/redact_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4253ee8dab999331"}},{"code_sha256_prefix":"89dc660e78f4dc9e","entry":"normalise_name","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/canonicalize_knowledge_points.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/canonicalize_knowledge_points.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89dc660e78f4dc9e"}},{"code_sha256_prefix":"10c3629b60f7125a","entry":"normalise_name_loose","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/canonicalize_knowledge_points.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/canonicalize_knowledge_points.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10c3629b60f7125a"}},{"code_sha256_prefix":"8c447893766aecd7","entry":"normalise_text","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/build_textbook_fragments.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/build_textbook_fragments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c447893766aecd7"}},{"code_sha256_prefix":"485b24e15866830e","entry":"output_dir_for","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"corpus_scripts/mineru_parse.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/corpus_scripts/mineru_parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"485b24e15866830e"}},{"code_sha256_prefix":"301abe73b030e5eb","entry":"output_for","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"corpus_scripts/ocr_fix.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/corpus_scripts/ocr_fix.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"301abe73b030e5eb"}},{"code_sha256_prefix":"ff323da75a3b8154","entry":"parent_key","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/build_targeted_guidance.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/build_targeted_guidance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff323da75a3b8154"}},{"code_sha256_prefix":"7b48b0a666cd173a","entry":"parents_of","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/build_kp_guidance.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/build_kp_guidance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b48b0a666cd173a"}},{"code_sha256_prefix":"a1edcbf8c001fd15","entry":"parse_latex","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"eval/judge.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/eval/judge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1edcbf8c001fd15"}},{"code_sha256_prefix":"827fdd17ad5f2006","entry":"read_json","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/build_kp_guidance.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/build_kp_guidance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"827fdd17ad5f2006"}},{"code_sha256_prefix":"c12a8c71b4ffe271","entry":"read_json","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/build_targeted_guidance.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/build_targeted_guidance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c12a8c71b4ffe271"}},{"code_sha256_prefix":"024ac62b35bdff98","entry":"readjsonl2list","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"eval/judge_utils.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/eval/judge_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"024ac62b35bdff98"}},{"code_sha256_prefix":"8a2c534258845b4f","entry":"resolve_book","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/build_textbook_fragments.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/build_textbook_fragments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a2c534258845b4f"}},{"code_sha256_prefix":"abe9bb810ab211c3","entry":"sub_key","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"filter/build_targeted_guidance.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/filter/build_targeted_guidance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"abe9bb810ab211c3"}},{"code_sha256_prefix":"9c54c8badc2971a6","entry":"tokens_with_spans","repo":"thunlp/StudyBench","repo_kind":"found_in_text","path":"corpus_scripts/redact_eval.py","file_url":"https://github.com/thunlp/StudyBench/blob/HEAD/corpus_scripts/redact_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c54c8badc2971a6"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.AI","source":"arxiv_daily_20260902.json"},"syntology_extracted_results":null}