{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-art-of-saying-no-contextual-noncompliance","title":"The Art of Saying No: Contextual Noncompliance in Language Models","arxiv_id":"2407.12043","date":"2024-07-02","proceeding":null,"authors":["Faeze Brahman","Sachin Kumar","Vidhisha Balachandran","Pradeep Dasigi","Valentina Pyatkin","Abhilasha Ravichander","Sarah Wiegreffe","Nouha Dziri","Khyathi Chandu","Jack Hessel","Yulia Tsvetkov","Noah A. Smith","Yejin Choi","Hannaneh Hajishirzi"],"abstract":"Chat-based language models are designed to be helpful, yet they should not comply with every user request. While most existing work primarily focuses on refusal of \"unsafe\" queries, we posit that the scope of noncompliance should be broadened. We introduce a comprehensive taxonomy of contextual noncompliance describing when and how models should not comply with user requests. Our taxonomy spans a wide range of categories including incomplete, unsupported, indeterminate, and humanizing requests (in addition to unsafe requests). To test noncompliance capabilities of language models, we use this taxonomy to develop a new evaluation suite of 1000 noncompliance prompts. We find that most existing models show significantly high compliance rates in certain previously understudied categories with models like GPT-4 incorrectly complying with as many as 30% of requests. To address these gaps, we explore different training strategies using a synthetically-generated training set of requests and expected noncompliant responses. Our experiments demonstrate that while direct finetuning of instruction-tuned models can lead to both over-refusal and a decline in general capabilities, using parameter efficient methods like low rank adapters helps to strike a good balance between appropriate noncompliance and other capabilities.","url_abs":"https://arxiv.org/abs/2407.12043v2","url_pdf":"https://arxiv.org/pdf/2407.12043v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-art-of-saying-no-contextual-noncompliance","repo_url":"https://github.com/allenai/compred","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.12043","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.12043"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/allenai/compred","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":4},"by_repo_kind":{"listed":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"148dc2c38b7cd922","entry":"prepare_sample_text","repo":"allenai/compred","repo_kind":"listed","path":"sft.py","file_url":"https://github.com/allenai/compred/blob/HEAD/sft.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"148dc2c38b7cd922"}},{"code_sha256_prefix":"fed28117a0a0df7c","entry":"get_paired_dataset","repo":"allenai/compred","repo_kind":"listed","path":"dpo.py","file_url":"https://github.com/allenai/compred/blob/HEAD/dpo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fed28117a0a0df7c"}},{"code_sha256_prefix":"a6a916160dd1e328","entry":"prepare_sample_text_plain","repo":"allenai/compred","repo_kind":"listed","path":"sft.py","file_url":"https://github.com/allenai/compred/blob/HEAD/sft.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a6a916160dd1e328"}},{"code_sha256_prefix":"bcbebc92193d1a96","entry":"prepare_sample_text_subredditname","repo":"allenai/compred","repo_kind":"listed","path":"sft.py","file_url":"https://github.com/allenai/compred/blob/HEAD/sft.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bcbebc92193d1a96"}},{"code_sha256_prefix":"dffe35a176c72680","entry":"subsample","repo":"allenai/compred","repo_kind":"listed","path":"dpo.py","file_url":"https://github.com/allenai/compred/blob/HEAD/dpo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dffe35a176c72680"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}