{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/citekit-a-modular-toolkit-for-large-language","title":"Citekit: A Modular Toolkit for Large Language Model Citation Generation","arxiv_id":"2408.04662","date":"2024-08-06","proceeding":null,"authors":["Jiajun Shen","Tong Zhou","Yubo Chen","Kang Liu"],"abstract":"Enabling Large Language Models (LLMs) to generate citations in Question-Answering (QA) tasks is an emerging paradigm aimed at enhancing the verifiability of their responses when LLMs are utilizing external references to generate an answer. However, there is currently no unified framework to standardize and fairly compare different citation generation methods, leading to difficulties in reproducing different methods and a comprehensive assessment. To cope with the problems above, we introduce \\name, an open-source and modular toolkit designed to facilitate the implementation and evaluation of existing citation generation methods, while also fostering the development of new approaches to improve citation quality in LLM outputs. This tool is highly extensible, allowing users to utilize 4 main modules and 14 components to construct a pipeline, evaluating an existing method or innovative designs. Our experiments with two state-of-the-art LLMs and 11 citation generation baselines demonstrate varying strengths of different modules in answer accuracy and citation quality improvement, as well as the challenge of enhancing granularity. Based on our analysis of the effectiveness of components, we propose a new method, self-RAG \\snippet, obtaining a balanced answer accuracy and citation quality. Citekit is released at https://github.com/SjJ1017/Citekit.","url_abs":"https://arxiv.org/abs/2408.04662v2","url_pdf":"https://arxiv.org/pdf/2408.04662v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"citekit-a-modular-toolkit-for-large-language","repo_url":"https://github.com/sjj1017/citekit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"rag","task_name":"RAG"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2408.04662","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.04662"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sjj1017/citekit","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ece0969bd45a5478","entry":"compute_str_em","repo":"sjj1017/citekit","repo_kind":"official","path":"citekit/utils/utils.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/citekit/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ece0969bd45a5478"}},{"code_sha256_prefix":"7f4fb396d1f4c613","entry":"cut_and_make_as","repo":"sjj1017/citekit","repo_kind":"official","path":"methods/ALCE_interact.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/methods/ALCE_interact.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f4fb396d1f4c613"}},{"code_sha256_prefix":"1d50ff426c4bbbbd","entry":"get_cite","repo":"sjj1017/citekit","repo_kind":"official","path":"citekit/evaluator/evaluator.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/citekit/evaluator/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d50ff426c4bbbbd"}},{"code_sha256_prefix":"e11e7457d509e77d","entry":"gtr_build_index","repo":"sjj1017/citekit","repo_kind":"official","path":"citekit/cite_modules/Retrieve.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/citekit/cite_modules/Retrieve.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e11e7457d509e77d"}},{"code_sha256_prefix":"1e09632cc01ef34a","entry":"merge_str_dicts","repo":"sjj1017/citekit","repo_kind":"official","path":"citekit/pipeline/pipeline.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/citekit/pipeline/pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e09632cc01ef34a"}},{"code_sha256_prefix":"3cb5748b00b30bf8","entry":"one_paragraph","repo":"sjj1017/citekit","repo_kind":"official","path":"citekit/utils/utils.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/citekit/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3cb5748b00b30bf8"}},{"code_sha256_prefix":"87f7647683a35e3a","entry":"one_paragraph","repo":"sjj1017/citekit","repo_kind":"official","path":"methods/ALCE_interact.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/methods/ALCE_interact.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87f7647683a35e3a"}},{"code_sha256_prefix":"9fea39427c8a945f","entry":"strong_one_paragraph","repo":"sjj1017/citekit","repo_kind":"official","path":"citekit/utils/utils.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/citekit/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fea39427c8a945f"}},{"code_sha256_prefix":"53077c3e14202ce3","entry":"compute_mauve","repo":"sjj1017/citekit","repo_kind":"official","path":"citekit/evaluator/evaluator.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/citekit/evaluator/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53077c3e14202ce3"}},{"code_sha256_prefix":"99ba7cd8c943a75c","entry":"entail","repo":"sjj1017/citekit","repo_kind":"official","path":"citekit/evaluator/evaluator.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/citekit/evaluator/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99ba7cd8c943a75c"}},{"code_sha256_prefix":"2b1aa6c33e7b0d55","entry":"load_model","repo":"sjj1017/citekit","repo_kind":"official","path":"citekit/cite_modules/LLM.py","file_url":"https://github.com/sjj1017/citekit/blob/HEAD/citekit/cite_modules/LLM.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b1aa6c33e7b0d55"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}