{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/out-of-style-rag-s-fragility-to-linguistic","title":"Out of Style: RAG's Fragility to Linguistic Variation","arxiv_id":"2504.08231","date":"2025-04-11","proceeding":null,"authors":["Tianyu Cao","Neel Bhandari","Akhila Yerukola","Akari Asai","Maarten Sap"],"abstract":"Despite the impressive performance of Retrieval-augmented Generation (RAG) systems across various NLP benchmarks, their robustness in handling real-world user-LLM interaction queries remains largely underexplored. This presents a critical gap for practical deployment, where user queries exhibit greater linguistic variations and can trigger cascading errors across interdependent RAG components. In this work, we systematically analyze how varying four linguistic dimensions (formality, readability, politeness, and grammatical correctness) impact RAG performance. We evaluate two retrieval models and nine LLMs, ranging from 3 to 72 billion parameters, across four information-seeking Question Answering (QA) datasets. Our results reveal that linguistic reformulations significantly impact both retrieval and generation stages, leading to a relative performance drop of up to 40.41% in Recall@5 scores for less formal queries and 38.86% in answer match scores for queries containing grammatical errors. Notably, RAG systems exhibit greater sensitivity to such variations compared to LLM-only generations, highlighting their vulnerability to error propagation due to linguistic shifts. These findings highlight the need for improved robustness techniques to enhance reliability in diverse user interactions.","url_abs":"https://arxiv.org/abs/2504.08231v1","url_pdf":"https://arxiv.org/pdf/2504.08231v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"out-of-style-rag-s-fragility-to-linguistic","repo_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2504.08231","atlas_url":"https://app.syntology.ai/?focus=2504.08231","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.08231"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/springcty/rag-fragility-to-linguistic-variation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":11,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7f7c1acb9a28d150","entry":"check_included","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"LLM_generation/c_eval_generation.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/LLM_generation/c_eval_generation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f7c1acb9a28d150"}},{"code_sha256_prefix":"7b7841244f89b921","entry":"combine_texts","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"LLM_generation/a_vllm_generation.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/LLM_generation/a_vllm_generation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b7841244f89b921"}},{"code_sha256_prefix":"48e3241b4aec397e","entry":"count_syllables","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"query_rewriting/readability_rewriting.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/query_rewriting/readability_rewriting.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48e3241b4aec397e"}},{"code_sha256_prefix":"af8c6a9399918efd","entry":"edit_word","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"query_rewriting/grammar_conversion.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/query_rewriting/grammar_conversion.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af8c6a9399918efd"}},{"code_sha256_prefix":"76a0e7a7c54bbde5","entry":"em_score","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"LLM_generation/c_eval_generation.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/LLM_generation/c_eval_generation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76a0e7a7c54bbde5"}},{"code_sha256_prefix":"4c0e1b3b63e3f2cf","entry":"flesch_reading_ease","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"query_rewriting/readability_rewriting.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/query_rewriting/readability_rewriting.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c0e1b3b63e3f2cf"}},{"code_sha256_prefix":"afab3aee4c1d97a0","entry":"format_prompt","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"LLM_generation/utils/vllm_inference.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/LLM_generation/utils/vllm_inference.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afab3aee4c1d97a0"}},{"code_sha256_prefix":"ad66e7d5ebc3f6df","entry":"format_prompt_gemma","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"LLM_generation/utils/vllm_inference.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/LLM_generation/utils/vllm_inference.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad66e7d5ebc3f6df"}},{"code_sha256_prefix":"d74aa30b65ed63fd","entry":"load_data","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"LLM_generation/b_vllm_none_retrieval.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/LLM_generation/b_vllm_none_retrieval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d74aa30b65ed63fd"}},{"code_sha256_prefix":"33afe86db97be0bd","entry":"load_linguistic_query","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"LLM_generation/utils/few_shot_prompting.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/LLM_generation/utils/few_shot_prompting.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33afe86db97be0bd"}},{"code_sha256_prefix":"bf618b8796ac4942","entry":"normalize_answer","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"LLM_generation/c_eval_generation.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/LLM_generation/c_eval_generation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf618b8796ac4942"}},{"code_sha256_prefix":"bc619506cd82ac84","entry":"create_few_shot_examples","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"LLM_generation/utils/few_shot_prompting.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/LLM_generation/utils/few_shot_prompting.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc619506cd82ac84"}},{"code_sha256_prefix":"8ff350ea77c22ee9","entry":"format_nq","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"query_rewriting/grammar_conversion.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/query_rewriting/grammar_conversion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ff350ea77c22ee9"}},{"code_sha256_prefix":"c8439aa5de3482a9","entry":"load_dataset_samples","repo":"springcty/rag-fragility-to-linguistic-variation","repo_kind":"official","path":"query_rewriting/grammar_conversion.py","file_url":"https://github.com/springcty/rag-fragility-to-linguistic-variation/blob/HEAD/query_rewriting/grammar_conversion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8439aa5de3482a9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}