{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/enabling-large-language-models-to-generate","title":"Enabling Large Language Models to Generate Text with Citations","arxiv_id":"2305.14627","date":"2023-05-24","proceeding":null,"authors":["Tianyu Gao","Howard Yen","Jiatong Yu","Danqi Chen"],"abstract":"Large language models (LLMs) have emerged as a widely-used tool for information seeking, but their generated outputs are prone to hallucination. In this work, our aim is to allow LLMs to generate text with citations, improving their factual correctness and verifiability. Existing work mainly relies on commercial search engines and human evaluation, making it challenging to reproduce and compare different modeling approaches. We propose ALCE, the first benchmark for Automatic LLMs' Citation Evaluation. ALCE collects a diverse set of questions and retrieval corpora and requires building end-to-end systems to retrieve supporting evidence and generate answers with citations. We develop automatic metrics along three dimensions -- fluency, correctness, and citation quality -- and demonstrate their strong correlation with human judgements. Our experiments with state-of-the-art LLMs and novel prompting strategies show that current systems have considerable room for improvement -- For example, on the ELI5 dataset, even the best models lack complete citation support 50% of the time. Our analyses further highlight promising future directions, including developing better retrievers, advancing long-context LLMs, and improving the ability to synthesize information from multiple sources.","url_abs":"https://arxiv.org/abs/2305.14627v2","url_pdf":"https://arxiv.org/pdf/2305.14627v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"enabling-large-language-models-to-generate","repo_url":"https://github.com/princeton-nlp/alce","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[{"slug":"alce","name":"ALCE","full_name":"Automatic LLMs' Citation Evaluation"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.14627","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.14627"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-nlp/alce","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6fd0394cdc745c1a","entry":"doc_to_text_dense","repo":"princeton-nlp/alce","repo_kind":"official","path":"searcher.py","file_url":"https://github.com/princeton-nlp/alce/blob/HEAD/searcher.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6fd0394cdc745c1a"}},{"code_sha256_prefix":"af88e97a5d3a1410","entry":"doc_to_text_tfidf","repo":"princeton-nlp/alce","repo_kind":"official","path":"searcher.py","file_url":"https://github.com/princeton-nlp/alce/blob/HEAD/searcher.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af88e97a5d3a1410"}},{"code_sha256_prefix":"c7d29eb322d5d6e3","entry":"make_doc_prompt","repo":"princeton-nlp/alce","repo_kind":"official","path":"utils.py","file_url":"https://github.com/princeton-nlp/alce/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7d29eb322d5d6e3"}},{"code_sha256_prefix":"dae7ab386661a4f4","entry":"normalize_answer","repo":"princeton-nlp/alce","repo_kind":"official","path":"utils.py","file_url":"https://github.com/princeton-nlp/alce/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dae7ab386661a4f4"}},{"code_sha256_prefix":"5df8b525aada9853","entry":"remove_citations","repo":"princeton-nlp/alce","repo_kind":"official","path":"post_hoc_cite.py","file_url":"https://github.com/princeton-nlp/alce/blob/HEAD/post_hoc_cite.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5df8b525aada9853"}},{"code_sha256_prefix":"c15a972a302a288b","entry":"gtr_build_index","repo":"princeton-nlp/alce","repo_kind":"official","path":"retrieval.py","file_url":"https://github.com/princeton-nlp/alce/blob/HEAD/retrieval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c15a972a302a288b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}