{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-verification-improves-few-shot-clinical","title":"Self-Verification Improves Few-Shot Clinical Information Extraction","arxiv_id":"2306.00024","date":"2023-05-30","proceeding":null,"authors":["Zelalem Gero","Chandan Singh","Hao Cheng","Tristan Naumann","Michel Galley","Jianfeng Gao","Hoifung Poon"],"abstract":"Extracting patient information from unstructured text is a critical task in health decision-support and clinical research. Large language models (LLMs) have shown the potential to accelerate clinical curation via few-shot in-context learning, in contrast to supervised learning which requires much more costly human annotations. However, despite drastic advances in modern LLMs such as GPT-4, they still struggle with issues regarding accuracy and interpretability, especially in mission-critical domains such as health. Here, we explore a general mitigation framework using self-verification, which leverages the LLM to provide provenance for its own extraction and check its own outputs. This is made possible by the asymmetry between verification and generation, where the latter is often much easier than the former. Experimental results show that our method consistently improves accuracy for various LLMs in standard clinical information extraction tasks. Additionally, self-verification yields interpretations in the form of a short text span corresponding to each output, which makes it very efficient for human experts to audit the results, paving the way towards trustworthy extraction of clinical information in resource-constrained scenarios. To facilitate future research in this direction, we release our code and prompts.","url_abs":"https://arxiv.org/abs/2306.00024v1","url_pdf":"https://arxiv.org/pdf/2306.00024v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-verification-improves-few-shot-clinical","repo_url":"https://github.com/microsoft/clinical-self-verification","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.00024","atlas_url":"https://app.syntology.ai/?focus=2306.00024","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.00024"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/clinical-self-verification","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b9d6d8e6bcc7fcc0","entry":"colorize","repo":"microsoft/clinical-self-verification","repo_kind":"official","path":"clin/viz.py","file_url":"https://github.com/microsoft/clinical-self-verification/blob/HEAD/clin/viz.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9d6d8e6bcc7fcc0"}},{"code_sha256_prefix":"22d82a535b18911d","entry":"convert_keys_to_lowercase","repo":"microsoft/clinical-self-verification","repo_kind":"official","path":"clin/parse.py","file_url":"https://github.com/microsoft/clinical-self-verification/blob/HEAD/clin/parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22d82a535b18911d"}},{"code_sha256_prefix":"398ecdba4531d165","entry":"process_ebm_lists","repo":"microsoft/clinical-self-verification","repo_kind":"official","path":"clin/eval/ebm.py","file_url":"https://github.com/microsoft/clinical-self-verification/blob/HEAD/clin/eval/ebm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"398ecdba4531d165"}},{"code_sha256_prefix":"f1d6de721ed23fd0","entry":"sample_shots_excluding_i","repo":"microsoft/clinical-self-verification","repo_kind":"official","path":"clin/parse.py","file_url":"https://github.com/microsoft/clinical-self-verification/blob/HEAD/clin/parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1d6de721ed23fd0"}},{"code_sha256_prefix":"4139e67944d6519c","entry":"str_to_list","repo":"microsoft/clinical-self-verification","repo_kind":"official","path":"clin/parse.py","file_url":"https://github.com/microsoft/clinical-self-verification/blob/HEAD/clin/parse.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4139e67944d6519c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}