{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-13786","title":"Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions","arxiv_id":"2608.13786","date":"2026-08-13","proceeding":null,"authors":["Qingfang Liu","Qiao Jin","Joe D. Menke","Thorsten Kahnt","Zhiyong Lu"],"abstract":"Large language model (LLM) chatbots are increasingly used to answer clinical questions with citations to relevant clinical studies. Prior research has largely focused on citation fabrication, leaving a gap in evaluating the quality of retrieved studies and the factors driving their selection. In this study, we evaluated three general-purpose LLM chatbots: Claude Sonnet 5, Gemini 3.1 Pro, and ChatGPT GPT-5.5. We prompted the models with clinical questions adapted from 20 review questions in Issues 6 and 7 of the 2026 Cochrane Database of Systematic Reviews, simulating patient, clinician, and evidence-synthesis researcher roles. Each chatbot was queried under each user role with four independent repetitions, yielding 720 responses. Each chatbot was asked to support its answers with primary clinical citations, which we benchmarked against the included and excluded study sets of the Cochrane reviews. On average, a chatbot response retrieved 39.2% $\\pm$ 29.8% of Cochrane included studies, while citing 5.0% $\\pm$ 9.4% of excluded studies. Recall of Cochrane included studies varied significantly by model and user role. ChatGPT achieved higher recall than Claude or Gemini (63.1% $\\pm$ 29.5% vs. 37.0% $\\pm$ 23.8% vs. 17.3% $\\pm$ 13.1%; $p=2.0\\times10^{-5}$). The researcher role yielded higher recall than the clinician or patient roles (42.8% $\\pm$ 30.8% vs. 38.6% $\\pm$ 28.9% vs. 36.1% $\\pm$ 29.3%; $p=2.0\\times10^{-5}$). Controlling for publication year, citations per year, and open-access status, sample size was the only independently significant predictor of retrieval (odds ratio 1.80 per 1-unit increase in log sample size, 95% CI 1.37-2.36, $p=2.34\\times10^{-5}$). These findings suggest that while LLM chatbots can retrieve some studies identified by expert reviewers, their performance varies by model and user role, and they exhibit a bias toward clinical trials with larger sample sizes.","url_abs":"https://arxiv.org/abs/2608.13786","url_pdf":"https://arxiv.org/pdf/2608.13786","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2608.13786","atlas_url":"https://app.syntology.ai/?focus=2608.13786","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.13786"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias","reach":null}],"summary":{"ran":25,"unverified":6},"by_repo_kind":{"found_in_text":{"samples":31,"ran":25,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":31,"samples":[{"code_sha256_prefix":"2d0de3f9a753d906","entry":"calculate_mann_whitney_test","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"demo/overlap_metrics.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/demo/overlap_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d0de3f9a753d906"}},{"code_sha256_prefix":"0843b6bce84576ef","entry":"calculate_multi_set_jaccard","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"demo/overlap_metrics.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/demo/overlap_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0843b6bce84576ef"}},{"code_sha256_prefix":"68c30d932a7949aa","entry":"calculate_replicate_consistency","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"demo/overlap_metrics.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/demo/overlap_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68c30d932a7949aa"}},{"code_sha256_prefix":"25480657033da448","entry":"cell","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/reference_indexing_schema.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/reference_indexing_schema.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"25480657033da448"}},{"code_sha256_prefix":"a15ecee30950237e","entry":"citation_identity_note","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD000510/analyze_cd000510_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD000510/analyze_cd000510_roles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a15ecee30950237e"}},{"code_sha256_prefix":"d5c2b8646741a30a","entry":"collect_review_pdfs","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/cochrane_review_source.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/cochrane_review_source.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d5c2b8646741a30a"}},{"code_sha256_prefix":"cd382468e750cf17","entry":"compact","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/build_reference_indexing_from_cochrane_ris.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/build_reference_indexing_from_cochrane_ris.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd382468e750cf17"}},{"code_sha256_prefix":"744acc2017757c05","entry":"compact","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/reference_indexing_schema.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/reference_indexing_schema.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"744acc2017757c05"}},{"code_sha256_prefix":"31ff80b0e0d7f9b7","entry":"dimension_summaries","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"figures/make_paper_figures.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/figures/make_paper_figures.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31ff80b0e0d7f9b7"}},{"code_sha256_prefix":"a504f8f552b1365c","entry":"extract_claude_list","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD001452/analyze_cd001452_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD001452/analyze_cd001452_roles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a504f8f552b1365c"}},{"code_sha256_prefix":"274aeac953e50576","entry":"extract_terminal_list","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD007654/analyze_cd007654_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD007654/analyze_cd007654_roles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"274aeac953e50576"}},{"code_sha256_prefix":"c667163c27e05788","entry":"is_review_pdf_path","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/cochrane_review_source.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/cochrane_review_source.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c667163c27e05788"}},{"code_sha256_prefix":"9c02250cd6cbdd4a","entry":"last_heading_index","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD001452/analyze_cd001452_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD001452/analyze_cd001452_roles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c02250cd6cbdd4a"}},{"code_sha256_prefix":"f1b49bfb18520d40","entry":"load_matches","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"demo/build_demo.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/demo/build_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1b49bfb18520d40"}},{"code_sha256_prefix":"2effd1f8b982f74e","entry":"load_model_metadata","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"demo/build_demo.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/demo/build_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2effd1f8b982f74e"}},{"code_sha256_prefix":"62eb804824fdba32","entry":"load_role_labels","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"demo/build_demo.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/demo/build_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62eb804824fdba32"}},{"code_sha256_prefix":"4ebd7f5545fabcde","entry":"normalize","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD000510/analyze_cd000510_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD000510/analyze_cd000510_roles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ebd7f5545fabcde"}},{"code_sha256_prefix":"d6054082d8efea8b","entry":"normalize","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD005354/analyze_cd005354_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD005354/analyze_cd005354_roles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6054082d8efea8b"}},{"code_sha256_prefix":"e9816484b1b198c7","entry":"normalize","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD007654/analyze_cd007654_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD007654/analyze_cd007654_roles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9816484b1b198c7"}},{"code_sha256_prefix":"20c568d253ca23b7","entry":"normalize_text","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/build_reference_indexing_from_cochrane_ris.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/build_reference_indexing_from_cochrane_ris.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20c568d253ca23b7"}},{"code_sha256_prefix":"c900c29ec5e5f4c2","entry":"resolve_candidate","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD000510/analyze_cd000510_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD000510/analyze_cd000510_roles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c900c29ec5e5f4c2"}},{"code_sha256_prefix":"90f97f1529633b9d","entry":"review_collection_name","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/cochrane_review_source.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/cochrane_review_source.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90f97f1529633b9d"}},{"code_sha256_prefix":"6eae04cab3d1e143","entry":"ris_records","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD005354/analyze_cd005354_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD005354/analyze_cd005354_roles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6eae04cab3d1e143"}},{"code_sha256_prefix":"a25acaf2819e6f17","entry":"title_tokens","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/build_reference_indexing_from_cochrane_ris.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/build_reference_indexing_from_cochrane_ris.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a25acaf2819e6f17"}},{"code_sha256_prefix":"9ac2ddedf8b5c895","entry":"unique_preserve_order","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/reference_indexing_schema.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/reference_indexing_schema.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ac2ddedf8b5c895"}},{"code_sha256_prefix":"e8edea7b748fea72","entry":"pmids_to_pmc_info","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/pubmed_utils.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/pubmed_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8edea7b748fea72"}},{"code_sha256_prefix":"d300858d9f7ff348","entry":"pubmed_efetch_details","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/pubmed_utils.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/pubmed_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d300858d9f7ff348"}},{"code_sha256_prefix":"c68386fb79c34681","entry":"pubmed_esearch","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"benchmark_tools/pubmed_utils.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/benchmark_tools/pubmed_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c68386fb79c34681"}},{"code_sha256_prefix":"e07b0c7b9ac66d23","entry":"resolve_candidate","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD007654/analyze_cd007654_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD007654/analyze_cd007654_roles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e07b0c7b9ac66d23"}},{"code_sha256_prefix":"a4f47fcd6ecf26a1","entry":"review_clustered_bootstrap_intervals","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"figures/make_paper_figures.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/figures/make_paper_figures.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4f47fcd6ecf26a1"}},{"code_sha256_prefix":"cfff32b3ef1e3c4e","entry":"zip_text","repo":"QingfangLiu/llm-evidence-retrieval-bias","repo_kind":"found_in_text","path":"reviews/CD005354/analyze_cd005354_roles.py","file_url":"https://github.com/QingfangLiu/llm-evidence-retrieval-bias/blob/HEAD/reviews/CD005354/analyze_cd005354_roles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfff32b3ef1e3c4e"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}