{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dola-decoding-by-contrasting-layers-improves","title":"DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models","arxiv_id":"2309.03883","date":"2023-09-07","proceeding":null,"authors":["Yung-Sung Chuang","Yujia Xie","Hongyin Luo","Yoon Kim","James Glass","Pengcheng He"],"abstract":"Despite their impressive capabilities, large language models (LLMs) are prone to hallucinations, i.e., generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs that does not require conditioning on retrieved external knowledge nor additional fine-tuning. Our approach obtains the next-token distribution by contrasting the differences in logits obtained from projecting the later layers versus earlier layers to the vocabulary space, exploiting the fact that factual knowledge in an LLMs has generally been shown to be localized to particular transformer layers. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts. DoLa consistently improves the truthfulness across multiple choices tasks and open-ended generation tasks, for example improving the performance of LLaMA family models on TruthfulQA by 12-17% absolute points, demonstrating its potential in making LLMs reliably generate truthful facts.","url_abs":"https://arxiv.org/abs/2309.03883v2","url_pdf":"https://arxiv.org/pdf/2309.03883v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dola-decoding-by-contrasting-layers-improves","repo_url":"https://github.com/voidism/dola","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"dola-decoding-by-contrasting-layers-improves","repo_url":"https://github.com/JayZhang42/SLED","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}},{"paper_slug":"dola-decoding-by-contrasting-layers-improves","repo_url":"https://github.com/githubhyz/delta","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"dola-decoding-by-contrasting-layers-improves","repo_url":"https://github.com/shikiw/opera","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"truthfulqa","task_name":"TruthfulQA"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.03883","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.03883"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shikiw/opera","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/voidism/dola","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/githubhyz/delta","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JayZhang42/SLED","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"e82e04c183ec59da","entry":"LLamaQaStoppingCriteria","repo":"voidism/dola","repo_kind":"official","path":"dola.py","file_url":"https://github.com/voidism/dola/blob/HEAD/dola.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e82e04c183ec59da"}},{"code_sha256_prefix":"6da96069a08d8337","entry":"MaxNewTokensCriteria","repo":"voidism/dola","repo_kind":"official","path":"dola.py","file_url":"https://github.com/voidism/dola/blob/HEAD/dola.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6da96069a08d8337"}},{"code_sha256_prefix":"6874e02d86e35a49","entry":"StoppingCriteriaList","repo":"voidism/dola","repo_kind":"official","path":"dola.py","file_url":"https://github.com/voidism/dola/blob/HEAD/dola.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6874e02d86e35a49"}},{"code_sha256_prefix":"96fe095fcc6b821a","entry":"extract_answer_from_output","repo":"voidism/DoLa","repo_kind":"official","path":"factor_eval.py","file_url":"https://github.com/voidism/DoLa/blob/HEAD/factor_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96fe095fcc6b821a"}},{"code_sha256_prefix":"78392b8414df821e","entry":"load_csv","repo":"voidism/DoLa","repo_kind":"official","path":"factor_eval.py","file_url":"https://github.com/voidism/DoLa/blob/HEAD/factor_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"78392b8414df821e"}},{"code_sha256_prefix":"eed02985e4e3e50a","entry":"DoLa","repo":"voidism/dola","repo_kind":"official","path":"dola.py","file_url":"https://github.com/voidism/dola/blob/HEAD/dola.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eed02985e4e3e50a"}},{"code_sha256_prefix":"0f14b819bc361c25","entry":"download_url","repo":"voidism/DoLa","repo_kind":"official","path":"factor_eval.py","file_url":"https://github.com/voidism/DoLa/blob/HEAD/factor_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f14b819bc361c25"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}