{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/characterizing-llm-abstention-behavior-in","title":"Characterizing LLM Abstention Behavior in Science QA with Context Perturbations","arxiv_id":"2404.12452","date":"2024-04-18","proceeding":null,"authors":["Bingbing Wen","Bill Howe","Lucy Lu Wang"],"abstract":"The correct model response in the face of uncertainty is to abstain from answering a question so as not to mislead the user. In this work, we study the ability of LLMs to abstain from answering context-dependent science questions when provided insufficient or incorrect context. We probe model sensitivity in several settings: removing gold context, replacing gold context with irrelevant context, and providing additional context beyond what is given. In experiments on four QA datasets with six LLMs, we show that performance varies greatly across models, across the type of context provided, and also by question type; in particular, many LLMs seem unable to abstain from answering boolean questions using standard QA prompts. Our analysis also highlights the unexpected impact of abstention performance on QA task accuracy. Counter-intuitively, in some settings, replacing gold context with irrelevant context or adding irrelevant context to gold context can improve abstention performance in a way that results in improvements in task performance. Our results imply that changes are needed in QA dataset design and evaluation to more effectively assess the correctness and downstream impacts of model abstention.","url_abs":"https://arxiv.org/abs/2404.12452v2","url_pdf":"https://arxiv.org/pdf/2404.12452v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"characterizing-llm-abstention-behavior-in","repo_url":"https://github.com/bbwen/llm_scienceqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.12452","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.12452"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bbwen/llm_scienceqa","reach":null}],"summary":{"ran_draft_wrong":1,"ran_honours":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"d6eb7fc276a790c2","entry":"normalize_answer","repo":"bbwen/llm_scienceqa","repo_kind":"official","path":"evaluation/qasper_evaluation.py","file_url":"https://github.com/bbwen/llm_scienceqa/blob/HEAD/evaluation/qasper_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6eb7fc276a790c2"}},{"code_sha256_prefix":"e86e70615f8d4649","entry":"paragraph_f1_score","repo":"bbwen/llm_scienceqa","repo_kind":"official","path":"evaluation/qasper_evaluation.py","file_url":"https://github.com/bbwen/llm_scienceqa/blob/HEAD/evaluation/qasper_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e86e70615f8d4649"}},{"code_sha256_prefix":"5f8f36a4b206aba8","entry":"token_f1_score","repo":"bbwen/llm_scienceqa","repo_kind":"official","path":"evaluation/qasper_evaluation.py","file_url":"https://github.com/bbwen/llm_scienceqa/blob/HEAD/evaluation/qasper_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f8f36a4b206aba8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}