{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ralle-a-framework-for-developing-and","title":"RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models","arxiv_id":"2308.10633","date":"2023-08-21","proceeding":null,"authors":["Yasuto Hoshi","Daisuke Miyashita","Youyang Ng","Kento Tatsuno","Yasuhiro Morioka","Osamu Torii","Jun Deguchi"],"abstract":"Retrieval-augmented large language models (R-LLMs) combine pre-trained large language models (LLMs) with information retrieval systems to improve the accuracy of factual question-answering. However, current libraries for building R-LLMs provide high-level abstractions without sufficient transparency for evaluating and optimizing prompts within specific inference processes such as retrieval and generation. To address this gap, we present RaLLe, an open-source framework designed to facilitate the development, evaluation, and optimization of R-LLMs for knowledge-intensive tasks. With RaLLe, developers can easily develop and evaluate R-LLMs, improving hand-crafted prompts, assessing individual inference processes, and objectively measuring overall system performance quantitatively. By leveraging these features, developers can enhance the performance and accuracy of their R-LLMs in knowledge-intensive generation tasks. We open-source our code at https://github.com/yhoshi3/RaLLe.","url_abs":"https://arxiv.org/abs/2308.10633v2","url_pdf":"https://arxiv.org/pdf/2308.10633v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ralle-a-framework-for-developing-and","repo_url":"https://github.com/yhoshi3/ralle","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.10633","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.10633"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yhoshi3/ralle","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/yhoshi3/RaLLe","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_draft_wrong":1,"ran_fixture":1,"unverified":8},"by_repo_kind":{"official":{"samples":15,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e4462ed511216842","entry":"average_pool","repo":"yhoshi3/ralle","repo_kind":"official","path":"ralle/retrievers/embeddings/encode_functions/e5.py","file_url":"https://github.com/yhoshi3/ralle/blob/HEAD/ralle/retrievers/embeddings/encode_functions/e5.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4462ed511216842"}},{"code_sha256_prefix":"f06db4cb088e9ddf","entry":"flex_format","repo":"yhoshi3/ralle","repo_kind":"official","path":"ralle/utils.py","file_url":"https://github.com/yhoshi3/ralle/blob/HEAD/ralle/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f06db4cb088e9ddf"}},{"code_sha256_prefix":"5fa53ca5d798ad2d","entry":"flex_format_batch","repo":"yhoshi3/ralle","repo_kind":"official","path":"ralle/evaluation/evaluate.py","file_url":"https://github.com/yhoshi3/ralle/blob/HEAD/ralle/evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fa53ca5d798ad2d"}},{"code_sha256_prefix":"0f688c8a96ac6cce","entry":"initialize_faiss","repo":"yhoshi3/RaLLe","repo_kind":"official","path":"ralle/retrievers/initialize.py","file_url":"https://github.com/yhoshi3/RaLLe/blob/HEAD/ralle/retrievers/initialize.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f688c8a96ac6cce"}},{"code_sha256_prefix":"e50cb67e7f317872","entry":"initialize_vector_index","repo":"yhoshi3/RaLLe","repo_kind":"official","path":"ralle/retrievers/initialize.py","file_url":"https://github.com/yhoshi3/RaLLe/blob/HEAD/ralle/retrievers/initialize.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e50cb67e7f317872"}},{"code_sha256_prefix":"135db0e7ca9f6697","entry":"interpret_prompt","repo":"yhoshi3/ralle","repo_kind":"official","path":"ralle/utils.py","file_url":"https://github.com/yhoshi3/ralle/blob/HEAD/ralle/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"135db0e7ca9f6697"}},{"code_sha256_prefix":"233f1c06801ce83f","entry":"read_config","repo":"yhoshi3/ralle","repo_kind":"official","path":"ralle/utils.py","file_url":"https://github.com/yhoshi3/ralle/blob/HEAD/ralle/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"233f1c06801ce83f"}},{"code_sha256_prefix":"fae9ccd49478bc2c","entry":"Retriever","repo":"yhoshi3/RaLLe","repo_kind":"official","path":"ralle/retrievers/initialize.py","file_url":"https://github.com/yhoshi3/RaLLe/blob/HEAD/ralle/retrievers/initialize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fae9ccd49478bc2c"}},{"code_sha256_prefix":"24ced6d0e7e56049","entry":"encode","repo":"yhoshi3/ralle","repo_kind":"official","path":"ralle/retrievers/embeddings/encode_functions/e5.py","file_url":"https://github.com/yhoshi3/ralle/blob/HEAD/ralle/retrievers/embeddings/encode_functions/e5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24ced6d0e7e56049"}},{"code_sha256_prefix":"a1ffc0eab16b3711","entry":"initialize_bm25","repo":"yhoshi3/RaLLe","repo_kind":"official","path":"ralle/retrievers/initialize.py","file_url":"https://github.com/yhoshi3/RaLLe/blob/HEAD/ralle/retrievers/initialize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1ffc0eab16b3711"}},{"code_sha256_prefix":"7cdfbd35dd4dd689","entry":"initialize_corpus","repo":"yhoshi3/ralle","repo_kind":"official","path":"ralle/retrievers/corpus/initialize.py","file_url":"https://github.com/yhoshi3/ralle/blob/HEAD/ralle/retrievers/corpus/initialize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cdfbd35dd4dd689"}},{"code_sha256_prefix":"11aabe46259db0a9","entry":"initialize_diskann","repo":"yhoshi3/RaLLe","repo_kind":"official","path":"ralle/retrievers/initialize.py","file_url":"https://github.com/yhoshi3/RaLLe/blob/HEAD/ralle/retrievers/initialize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11aabe46259db0a9"}},{"code_sha256_prefix":"10ce681141110591","entry":"initialize_llm","repo":"yhoshi3/ralle","repo_kind":"official","path":"ralle/llms/initialize.py","file_url":"https://github.com/yhoshi3/ralle/blob/HEAD/ralle/llms/initialize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10ce681141110591"}},{"code_sha256_prefix":"2e70586c29bf1ca2","entry":"initialize_query_encoder","repo":"yhoshi3/ralle","repo_kind":"official","path":"ralle/retrievers/embeddings/initialize.py","file_url":"https://github.com/yhoshi3/ralle/blob/HEAD/ralle/retrievers/embeddings/initialize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e70586c29bf1ca2"}},{"code_sha256_prefix":"e17d224b96a47657","entry":"initialize_retriever","repo":"yhoshi3/ralle","repo_kind":"official","path":"ralle/retrievers/initialize.py","file_url":"https://github.com/yhoshi3/ralle/blob/HEAD/ralle/retrievers/initialize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e17d224b96a47657"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}