{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/openscholar-synthesizing-scientific","title":"OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs","arxiv_id":"2411.14199","date":"2024-11-21","proceeding":null,"authors":["Akari Asai","Jacqueline He","Rulin Shao","Weijia Shi","Amanpreet Singh","Joseph Chee Chang","Kyle Lo","Luca Soldaini","Sergey Feldman","Mike D'Arcy","David Wadden","Matt Latzke","Minyang Tian","Pan Ji","Shengyan Liu","Hao Tong","Bohao Wu","Yanyu Xiong","Luke Zettlemoyer","Graham Neubig","Dan Weld","Doug Downey","Wen-tau Yih","Pang Wei Koh","Hannaneh Hajishirzi"],"abstract":"Scientific progress depends on researchers' ability to synthesize the growing body of literature. Can large language models (LMs) assist scientists in this task? We introduce OpenScholar, a specialized retrieval-augmented LM that answers scientific queries by identifying relevant passages from 45 million open-access papers and synthesizing citation-backed responses. To evaluate OpenScholar, we develop ScholarQABench, the first large-scale multi-domain benchmark for literature search, comprising 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience, and biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o hallucinates citations 78 to 90% of the time, OpenScholar achieves citation accuracy on par with human experts. OpenScholar's datastore, retriever, and self-feedback inference loop also improves off-the-shelf LMs: for instance, OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's 32%. We open-source all of our code, models, datastore, data and a public demo.","url_abs":"https://arxiv.org/abs/2411.14199v1","url_pdf":"https://arxiv.org/pdf/2411.14199v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"openscholar-synthesizing-scientific","repo_url":"https://github.com/akariasai/openscholar","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"openscholar-synthesizing-scientific","repo_url":"https://github.com/akariasai/openscholar_experteval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"openscholar-synthesizing-scientific","repo_url":"https://github.com/akariasai/scholarbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.14199","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.14199"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/akariasai/openscholar","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/akariasai/scholarbench","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/akariasai/openscholar_experteval","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran_draft_wrong":3,"ran_violates":1,"unverified":7},"by_repo_kind":{"official":{"samples":12,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"ebcbc14c5f0a26eb","entry":"calculate_openai_api_cost","repo":"akariasai/openscholar","repo_kind":"official","path":"src/open_scholar.py","file_url":"https://github.com/akariasai/openscholar/blob/HEAD/src/open_scholar.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ebcbc14c5f0a26eb"}},{"code_sha256_prefix":"2254a963dae027e0","entry":"get_passage_pos_ids","repo":"akariasai/openscholar","repo_kind":"official","path":"retriever/api/passage_utils.py","file_url":"https://github.com/akariasai/openscholar/blob/HEAD/retriever/api/passage_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2254a963dae027e0"}},{"code_sha256_prefix":"4419dd4c768a4bb5","entry":"is_integer_string","repo":"akariasai/openscholar_experteval","repo_kind":"official","path":"use_search_apis.py","file_url":"https://github.com/akariasai/openscholar_experteval/blob/HEAD/use_search_apis.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4419dd4c768a4bb5"}},{"code_sha256_prefix":"5df8b525aada9853","entry":"remove_citations","repo":"akariasai/openscholar","repo_kind":"official","path":"src/open_scholar.py","file_url":"https://github.com/akariasai/openscholar/blob/HEAD/src/open_scholar.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5df8b525aada9853"}},{"code_sha256_prefix":"9fcb6754cab0d0dd","entry":"rerank_paragraphs_bge","repo":"akariasai/openscholar","repo_kind":"official","path":"src/open_scholar.py","file_url":"https://github.com/akariasai/openscholar/blob/HEAD/src/open_scholar.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9fcb6754cab0d0dd"}},{"code_sha256_prefix":"0c17dfce015ab2fe","entry":"calculate_avg_fine_grained_scores","repo":"akariasai/openscholar_experteval","repo_kind":"official","path":"compute_metrics.py","file_url":"https://github.com/akariasai/openscholar_experteval/blob/HEAD/compute_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c17dfce015ab2fe"}},{"code_sha256_prefix":"8c75664adcd9b212","entry":"get_acceptance_results","repo":"akariasai/openscholar_experteval","repo_kind":"official","path":"compute_metrics.py","file_url":"https://github.com/akariasai/openscholar_experteval/blob/HEAD/compute_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c75664adcd9b212"}},{"code_sha256_prefix":"e09711ced33247ae","entry":"get_finegrained_results","repo":"akariasai/openscholar_experteval","repo_kind":"official","path":"compute_metrics.py","file_url":"https://github.com/akariasai/openscholar_experteval/blob/HEAD/compute_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e09711ced33247ae"}},{"code_sha256_prefix":"f4d87e94b7582645","entry":"get_paper_data","repo":"akariasai/openscholar_experteval","repo_kind":"official","path":"use_search_apis.py","file_url":"https://github.com/akariasai/openscholar_experteval/blob/HEAD/use_search_apis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f4d87e94b7582645"}},{"code_sha256_prefix":"3ae3bb73bff5368f","entry":"normalize_citations","repo":"akariasai/openscholar_experteval","repo_kind":"official","path":"convert_output_files.py","file_url":"https://github.com/akariasai/openscholar_experteval/blob/HEAD/convert_output_files.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ae3bb73bff5368f"}},{"code_sha256_prefix":"5fd3281e2c36928f","entry":"remove_citations","repo":"akariasai/openscholar_experteval","repo_kind":"official","path":"convert_output_files.py","file_url":"https://github.com/akariasai/openscholar_experteval/blob/HEAD/convert_output_files.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5fd3281e2c36928f"}},{"code_sha256_prefix":"b6c21051ff3f26c8","entry":"search_paper_via_title","repo":"akariasai/openscholar_experteval","repo_kind":"official","path":"use_search_apis.py","file_url":"https://github.com/akariasai/openscholar_experteval/blob/HEAD/use_search_apis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6c21051ff3f26c8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}