{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lookback-lens-detecting-and-mitigating","title":"Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps","arxiv_id":"2407.07071","date":"2024-07-09","proceeding":null,"authors":["Yung-Sung Chuang","Linlu Qiu","Cheng-Yu Hsieh","Ranjay Krishna","Yoon Kim","James Glass"],"abstract":"When asked to summarize articles or answer questions given a passage, large language models (LLMs) can hallucinate details and respond with unsubstantiated answers that are inaccurate with respect to the input context. This paper describes a simple approach for detecting such contextual hallucinations. We hypothesize that contextual hallucinations are related to the extent to which an LLM attends to information in the provided context versus its own generations. Based on this intuition, we propose a simple hallucination detection model whose input features are given by the ratio of attention weights on the context versus newly generated tokens (for each attention head). We find that a linear classifier based on these lookback ratio features is as effective as a richer detector that utilizes the entire hidden states of an LLM or a text-based entailment model. The lookback ratio-based detector -- Lookback Lens -- is found to transfer across tasks and even models, allowing a detector that is trained on a 7B model to be applied (without retraining) to a larger 13B model. We further apply this detector to mitigate contextual hallucinations, and find that a simple classifier-guided decoding approach is able to reduce the amount of hallucination, for example by 9.6% in the XSum summarization task.","url_abs":"https://arxiv.org/abs/2407.07071v2","url_pdf":"https://arxiv.org/pdf/2407.07071v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lookback-lens-detecting-and-mitigating","repo_url":"https://github.com/voidism/lookback-lens","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"articles","task_name":"Articles"},{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.07071","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.07071"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/lm-sys/FastChat","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/voidism/lookback-lens","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/voidism/Lookback-Lens","reach":{"status":"ok"}}],"summary":{"ran":4,"ran_fixture":1,"unverified":15},"by_repo_kind":{"official":{"samples":20,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":20,"samples":[{"code_sha256_prefix":"10af7cdbc53df7e5","entry":"best_subspan_em","repo":"voidism/lookback-lens","repo_kind":"official","path":"eval_exact_match.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/eval_exact_match.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10af7cdbc53df7e5"}},{"code_sha256_prefix":"f10a909a93d528cf","entry":"convert_to_token_level","repo":"voidism/lookback-lens","repo_kind":"official","path":"step03_lookback_lens.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/step03_lookback_lens.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f10a909a93d528cf"}},{"code_sha256_prefix":"e6ade772c4fc8437","entry":"extract_time_series_features","repo":"voidism/Lookback-Lens","repo_kind":"official","path":"step03_lookback_lens.py","file_url":"https://github.com/voidism/Lookback-Lens/blob/HEAD/step03_lookback_lens.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6ade772c4fc8437"}},{"code_sha256_prefix":"746e55bc216a6a60","entry":"load_nq_open","repo":"voidism/lookback-lens","repo_kind":"official","path":"eval_exact_match.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/eval_exact_match.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"746e55bc216a6a60"}},{"code_sha256_prefix":"491bdf9d170e0f42","entry":"load_summarization","repo":"voidism/lookback-lens","repo_kind":"official","path":"step02_eval_gpt4o.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/step02_eval_gpt4o.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"491bdf9d170e0f42"}},{"code_sha256_prefix":"37a5eed2dbd663ca","entry":"gelu_fast","repo":"voidism/lookback-lens","repo_kind":"official","path":"transformers-4.32.0/src/transformers/activations_tf.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/transformers-4.32.0/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"37a5eed2dbd663ca"}},{"code_sha256_prefix":"b176efb26f72e507","entry":"get_architectures_from_config_class","repo":"voidism/lookback-lens","repo_kind":"official","path":"transformers-4.32.0/utils/create_dummy_models.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/transformers-4.32.0/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b176efb26f72e507"}},{"code_sha256_prefix":"3d999b95009c7537","entry":"get_config_class_from_processor_class","repo":"voidism/lookback-lens","repo_kind":"official","path":"transformers-4.32.0/utils/create_dummy_models.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/transformers-4.32.0/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d999b95009c7537"}},{"code_sha256_prefix":"0c0b81b081919d8e","entry":"get_processor_types_from_config_class","repo":"voidism/lookback-lens","repo_kind":"official","path":"transformers-4.32.0/utils/create_dummy_models.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/transformers-4.32.0/utils/create_dummy_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c0b81b081919d8e"}},{"code_sha256_prefix":"93fb30bbe389d023","entry":"load_jsonl","repo":"voidism/lookback-lens","repo_kind":"official","path":"step02_eval_gpt4o.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/step02_eval_gpt4o.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93fb30bbe389d023"}},{"code_sha256_prefix":"124e3218c2d64455","entry":"load_nq_open","repo":"voidism/lookback-lens","repo_kind":"official","path":"step01_extract_attns.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/step01_extract_attns.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"124e3218c2d64455"}},{"code_sha256_prefix":"3abd09dee278805a","entry":"load_nq_open","repo":"voidism/lookback-lens","repo_kind":"official","path":"step02_eval_gpt4o.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/step02_eval_gpt4o.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3abd09dee278805a"}},{"code_sha256_prefix":"d10fb7dd76f75e3b","entry":"load_nq_open","repo":"voidism/lookback-lens","repo_kind":"official","path":"step04_run_decoding.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/step04_run_decoding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d10fb7dd76f75e3b"}},{"code_sha256_prefix":"cc8c8c3ebf0c343f","entry":"mish","repo":"voidism/lookback-lens","repo_kind":"official","path":"transformers-4.32.0/src/transformers/activations_tf.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/transformers-4.32.0/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc8c8c3ebf0c343f"}},{"code_sha256_prefix":"9706a9a0b961096b","entry":"normalize_answer","repo":"voidism/lookback-lens","repo_kind":"official","path":"eval_exact_match.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/eval_exact_match.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9706a9a0b961096b"}},{"code_sha256_prefix":"b3a67ee5030c2ffe","entry":"num_tokens_from_message","repo":"voidism/lookback-lens","repo_kind":"official","path":"step01_extract_attns.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/step01_extract_attns.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3a67ee5030c2ffe"}},{"code_sha256_prefix":"aba17adf1edf4960","entry":"num_tokens_from_message","repo":"voidism/lookback-lens","repo_kind":"official","path":"step04_run_decoding.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/step04_run_decoding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aba17adf1edf4960"}},{"code_sha256_prefix":"e2d56cb91f999bef","entry":"quick_gelu","repo":"voidism/lookback-lens","repo_kind":"official","path":"transformers-4.32.0/src/transformers/activations_tf.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/transformers-4.32.0/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2d56cb91f999bef"}},{"code_sha256_prefix":"00d2f6a6c8b0f564","entry":"truncate_message","repo":"voidism/lookback-lens","repo_kind":"official","path":"step01_extract_attns.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/step01_extract_attns.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00d2f6a6c8b0f564"}},{"code_sha256_prefix":"0eaabf1df4cdff46","entry":"truncate_message","repo":"voidism/lookback-lens","repo_kind":"official","path":"step04_run_decoding.py","file_url":"https://github.com/voidism/lookback-lens/blob/HEAD/step04_run_decoding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0eaabf1df4cdff46"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}