{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/domainrag-a-chinese-benchmark-for-evaluating","title":"DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation","arxiv_id":"2406.05654","date":"2024-06-09","proceeding":null,"authors":["Shuting Wang","Jiongnan Liu","Shiren Song","Jiehan Cheng","Yuqi Fu","Peidong Guo","Kun Fang","Yutao Zhu","Zhicheng Dou"],"abstract":"Retrieval-Augmented Generation (RAG) offers a promising solution to address various limitations of Large Language Models (LLMs), such as hallucination and difficulties in keeping up with real-time updates. This approach is particularly critical in expert and domain-specific applications where LLMs struggle to cover expert knowledge. Therefore, evaluating RAG models in such scenarios is crucial, yet current studies often rely on general knowledge sources like Wikipedia to assess the models' abilities in solving common-sense problems. In this paper, we evaluated LLMs by RAG settings in a domain-specific context, college enrollment. We identified six required abilities for RAG models, including the ability in conversational RAG, analyzing structural information, faithfulness to external knowledge, denoising, solving time-sensitive problems, and understanding multi-document interactions. Each ability has an associated dataset with shared corpora to evaluate the RAG models' performance. We evaluated popular LLMs such as Llama, Baichuan, ChatGLM, and GPT models. Experimental results indicate that existing closed-book LLMs struggle with domain-specific questions, highlighting the need for RAG models to solve expert problems. Moreover, there is room for RAG models to improve their abilities in comprehending conversational history, analyzing structural information, denoising, processing multi-document interactions, and faithfulness in expert knowledge. We expect future studies could solve these problems better.","url_abs":"https://arxiv.org/abs/2406.05654v2","url_pdf":"https://arxiv.org/pdf/2406.05654v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"domainrag-a-chinese-benchmark-for-evaluating","repo_url":"https://github.com/ShootingWong/DomainRAG","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"domainrag-a-chinese-benchmark-for-evaluating","repo_url":"https://github.com/ruc-nlpir/flashrag","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"general-knowledge","task_name":"General Knowledge"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.05654","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.05654"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruc-nlpir/flashrag","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ShootingWong/DomainRAG","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran":11},"by_repo_kind":{"official":{"samples":12,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"bda36ad0c44dbd74","entry":"collator","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/evaluation/tasks/Datasets.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/evaluation/tasks/Datasets.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bda36ad0c44dbd74"}},{"code_sha256_prefix":"abd810c2287f2f36","entry":"em","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/evaluation/Evaluator.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/evaluation/Evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"abd810c2287f2f36"}},{"code_sha256_prefix":"1b33d678e1809cd1","entry":"em_strict","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/evaluation/Evaluator.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/evaluation/Evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b33d678e1809cd1"}},{"code_sha256_prefix":"0569ac90f9042b0d","entry":"get_answer","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/models/llm_models.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/models/llm_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0569ac90f9042b0d"}},{"code_sha256_prefix":"ad0fc83e3ddfae96","entry":"get_answer","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/evaluation/prediction_results/gpt_eval.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/evaluation/prediction_results/gpt_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad0fc83e3ddfae96"}},{"code_sha256_prefix":"91292e0608245752","entry":"gpt_generate_str","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/models/llm_models.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/models/llm_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91292e0608245752"}},{"code_sha256_prefix":"2e867288439d6ac9","entry":"gpt_generate_str","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/evaluation/prediction_results/gpt_eval.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/evaluation/prediction_results/gpt_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e867288439d6ac9"}},{"code_sha256_prefix":"82caf5acc631d96f","entry":"load_jsonl","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/models/main_retrieval.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/models/main_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"82caf5acc631d96f"}},{"code_sha256_prefix":"6205dd62c2627dde","entry":"normalize_answer","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/evaluation/Evaluator.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/evaluation/Evaluator.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6205dd62c2627dde"}},{"code_sha256_prefix":"a0b7c401fe27624d","entry":"remove_space","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/evaluation/tasks/Datasets.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/evaluation/tasks/Datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a0b7c401fe27624d"}},{"code_sha256_prefix":"14fd7d0254bdc425","entry":"runProcess_gpt","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/models/llm_models.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/models/llm_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14fd7d0254bdc425"}},{"code_sha256_prefix":"950491e7e95875bb","entry":"runProcess_gpt","repo":"ShootingWong/DomainRAG","repo_kind":"official","path":"BCM/evaluation/prediction_results/gpt_eval.py","file_url":"https://github.com/ShootingWong/DomainRAG/blob/HEAD/BCM/evaluation/prediction_results/gpt_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"950491e7e95875bb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}