{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-ai-assistants-know-what-they-don-t-know","title":"Can AI Assistants Know What They Don't Know?","arxiv_id":"2401.13275","date":"2024-01-24","proceeding":null,"authors":["Qinyuan Cheng","Tianxiang Sun","Xiangyang Liu","Wenwei Zhang","Zhangyue Yin","ShiMin Li","Linyang Li","Zhengfu He","Kai Chen","Xipeng Qiu"],"abstract":"Recently, AI assistants based on large language models (LLMs) show surprising performance in many tasks, such as dialogue, solving math problems, writing code, and using tools. Although LLMs possess intensive world knowledge, they still make factual errors when facing some knowledge intensive tasks, like open-domain question answering. These untruthful responses from the AI assistant may cause significant risks in practical applications. We believe that an AI assistant's refusal to answer questions it does not know is a crucial method for reducing hallucinations and making the assistant truthful. Therefore, in this paper, we ask the question \"Can AI assistants know what they don't know and express them through natural language?\" To answer this question, we construct a model-specific \"I don't know\" (Idk) dataset for an assistant, which contains its known and unknown questions, based on existing open-domain question answering datasets. Then we align the assistant with its corresponding Idk dataset and observe whether it can refuse to answer its unknown questions after alignment. Experimental results show that after alignment with Idk datasets, the assistant can refuse to answer most its unknown questions. For questions they attempt to answer, the accuracy is significantly higher than before the alignment.","url_abs":"https://arxiv.org/abs/2401.13275v2","url_pdf":"https://arxiv.org/pdf/2401.13275v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-ai-assistants-know-what-they-don-t-know","repo_url":"https://github.com/openmoss/say-i-dont-know","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"math","task_name":"Math"},{"task_slug":"open-domain-question-answering","task_name":"Open-Domain Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.13275","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.13275"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openmoss/say-i-dont-know","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran_fixture":2,"ran":3,"unverified":6},"by_repo_kind":{"official":{"samples":12,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"12f6803880f9b17e","entry":"extract_anthropic_prompt","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/dpo/preference_datasets.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/dpo/preference_datasets.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"12f6803880f9b17e"}},{"code_sha256_prefix":"2f0a690c70d11c23","entry":"gather_log_probs","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/deepspeed-chat-ppo/ppo_trainer.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/deepspeed-chat-ppo/ppo_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f0a690c70d11c23"}},{"code_sha256_prefix":"10ab71f27ffd7de8","entry":"get_local_run_dir","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/dpo/utils.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/dpo/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10ab71f27ffd7de8"}},{"code_sha256_prefix":"f2144808e1d3beed","entry":"load_auto_model_from_config","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/Inference/infer_reward_model.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/Inference/infer_reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2144808e1d3beed"}},{"code_sha256_prefix":"c79211fc0225a27f","entry":"preference_loss","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/dpo/trainers.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/dpo/trainers.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c79211fc0225a27f"}},{"code_sha256_prefix":"1d22c4f1f1dac3e2","entry":"read_dialogs_from_file","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/Inference/infer_llama.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/Inference/infer_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d22c4f1f1dac3e2"}},{"code_sha256_prefix":"b6e833b2bb85ecde","entry":"format_tokens_triviaqa","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/Inference/infer_llama.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/Inference/infer_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6e833b2bb85ecde"}},{"code_sha256_prefix":"2bd78998fa0ba9ea","entry":"format_tokens_triviaqa_for_ppo_reward","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/Inference/infer_llama.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/Inference/infer_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2bd78998fa0ba9ea"}},{"code_sha256_prefix":"8ab43db002bf984d","entry":"get_local_dir","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/dpo/utils.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/dpo/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ab43db002bf984d"}},{"code_sha256_prefix":"e7ce4c19888c6df6","entry":"get_remote_file","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/dpo/utils.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/dpo/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7ce4c19888c6df6"}},{"code_sha256_prefix":"429d42a3bf925da5","entry":"log_init","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"src/deepspeed-chat-ppo/rlhf_engine.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/src/deepspeed-chat-ppo/rlhf_engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"429d42a3bf925da5"}},{"code_sha256_prefix":"16fa287de857029b","entry":"process","repo":"openmoss/say-i-dont-know","repo_kind":"official","path":"Idk_datasets/process_sft_data.py","file_url":"https://github.com/openmoss/say-i-dont-know/blob/HEAD/Idk_datasets/process_sft_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16fa287de857029b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}