{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/truth-is-universal-robust-detection-of-lies","title":"Truth is Universal: Robust Detection of Lies in LLMs","arxiv_id":"2407.12831","date":"2024-07-03","proceeding":null,"authors":["Lennart Bürger","Fred A. Hamprecht","Boaz Nadler"],"abstract":"Large Language Models (LLMs) have revolutionised natural language processing, exhibiting impressive human-like capabilities. In particular, LLMs are capable of \"lying\", knowingly outputting false statements. Hence, it is of interest and importance to develop methods to detect when LLMs lie. Indeed, several authors trained classifiers to detect LLM lies based on their internal model activations. However, other researchers showed that these classifiers may fail to generalise, for example to negated statements. In this work, we aim to develop a robust method to detect when an LLM is lying. To this end, we make the following key contributions: (i) We demonstrate the existence of a two-dimensional subspace, along which the activation vectors of true and false statements can be separated. Notably, this finding is universal and holds for various LLMs, including Gemma-7B, LLaMA2-13B, Mistral-7B and LLaMA3-8B. Our analysis explains the generalisation failures observed in previous studies and sets the stage for more robust lie detection; (ii) Building upon (i), we construct an accurate LLM lie detector. Empirically, our proposed classifier achieves state-of-the-art performance, attaining 94% accuracy in both distinguishing true from false factual statements and detecting lies generated in real-world scenarios.","url_abs":"https://arxiv.org/abs/2407.12831v2","url_pdf":"https://arxiv.org/pdf/2407.12831v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"truth-is-universal-robust-detection-of-lies","repo_url":"https://github.com/sciai-lab/truth_is_universal","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"truth-is-universal-robust-detection-of-lies","repo_url":"https://github.com/admantwerp/llm-liedetector-keyphrase","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.12831","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.12831"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/admantwerp/llm-liedetector-keyphrase","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sciai-lab/truth_is_universal","reach":null}],"summary":{"ran_draft_wrong":8,"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1},"listed":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aa06a832bd24f818","entry":"ccs_loss","repo":"sciai-lab/truth_is_universal","repo_kind":"official","path":"probes.py","file_url":"https://github.com/sciai-lab/truth_is_universal/blob/HEAD/probes.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa06a832bd24f818"}},{"code_sha256_prefix":"55f2fc414934d5f3","entry":"get_acts","repo":"sciai-lab/truth_is_universal","repo_kind":"official","path":"generate_acts.py","file_url":"https://github.com/sciai-lab/truth_is_universal/blob/HEAD/generate_acts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55f2fc414934d5f3"}},{"code_sha256_prefix":"08c02598838bee27","entry":"learn_polarity_direction","repo":"sciai-lab/truth_is_universal","repo_kind":"official","path":"probes.py","file_url":"https://github.com/sciai-lab/truth_is_universal/blob/HEAD/probes.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08c02598838bee27"}},{"code_sha256_prefix":"17bda33662476720","entry":"learn_polarity_direction","repo":"sciai-lab/truth_is_universal","repo_kind":"official","path":"probes.py","file_url":"https://github.com/sciai-lab/truth_is_universal/blob/HEAD/probes.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17bda33662476720"}},{"code_sha256_prefix":"b4abfd1c5e88ea9b","entry":"learn_polarity_direction","repo":"admantwerp/llm-liedetector-keyphrase","repo_kind":"listed","path":"probes.py","file_url":"https://github.com/admantwerp/llm-liedetector-keyphrase/blob/HEAD/probes.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b4abfd1c5e88ea9b"}},{"code_sha256_prefix":"5bb3fba7746084dd","entry":"learn_truth_directions","repo":"sciai-lab/truth_is_universal","repo_kind":"official","path":"probes.py","file_url":"https://github.com/sciai-lab/truth_is_universal/blob/HEAD/probes.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bb3fba7746084dd"}},{"code_sha256_prefix":"4d10e785c5cb14e9","entry":"learn_truth_directions","repo":"sciai-lab/truth_is_universal","repo_kind":"official","path":"probes.py","file_url":"https://github.com/sciai-lab/truth_is_universal/blob/HEAD/probes.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d10e785c5cb14e9"}},{"code_sha256_prefix":"9d431fe1ea2c9e63","entry":"learn_truth_directions","repo":"admantwerp/llm-liedetector-keyphrase","repo_kind":"listed","path":"probes.py","file_url":"https://github.com/admantwerp/llm-liedetector-keyphrase/blob/HEAD/probes.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d431fe1ea2c9e63"}},{"code_sha256_prefix":"f0cc63177ce8b7a1","entry":"load_statements","repo":"sciai-lab/truth_is_universal","repo_kind":"official","path":"generate_acts.py","file_url":"https://github.com/sciai-lab/truth_is_universal/blob/HEAD/generate_acts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0cc63177ce8b7a1"}},{"code_sha256_prefix":"29ede6116837cea7","entry":"TTPD","repo":"sciai-lab/truth_is_universal","repo_kind":"official","path":"probes.py","file_url":"https://github.com/sciai-lab/truth_is_universal/blob/HEAD/probes.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29ede6116837cea7"}},{"code_sha256_prefix":"7a22af0ab3cffdaf","entry":"TTPD","repo":"admantwerp/llm-liedetector-keyphrase","repo_kind":"listed","path":"probes.py","file_url":"https://github.com/admantwerp/llm-liedetector-keyphrase/blob/HEAD/probes.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a22af0ab3cffdaf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}