{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-16576","title":"On the Robustness of LLM-Based Dense Retrievers: A Systematic Analysis of Generalizability and Stability","arxiv_id":"2604.16576","date":"2026-04-17","proceeding":null,"authors":["Yongkang Li","Panagiotis Eustratiadis","Yixing Fan","Evangelos Kanoulas"],"abstract":"Decoder-only large language models (LLMs) are increasingly replacing BERT-style architectures as the backbone for dense retrieval, achieving substantial performance gains and broad adoption. However, the robustness of these LLM-based retrievers remains underexplored. In this paper, we present the first systematic study of the robustness of state-of-the-art open-source LLM-based dense retrievers from two complementary perspectives: generalizability and stability. For generalizability, we evaluate retrieval effectiveness across four benchmarks spanning 30 datasets, using linear mixed-effects models to estimate marginal mean performance and disentangle intrinsic model capability from dataset heterogeneity. Our analysis reveals that while instruction-tuned models generally excel, those optimized for complex reasoning often suffer a ``specialization tax,'' exhibiting limited generalizability in broader contexts. For stability, we assess model resilience against both unintentional query variations~(e.g., paraphrasing, typos) and malicious adversarial attacks~(e.g., corpus poisoning). We find that LLM-based retrievers show improved robustness against typos and corpus poisoning compared to encoder-only baselines, yet remain vulnerable to semantic perturbations like synonymizing. Further analysis shows that embedding geometry (e.g., angular uniformity) provides predictive signals for lexical stability and suggests that scaling model size generally improves robustness. These findings inform future robustness-aware retriever design and principled benchmarking. Our code is publicly available at https://github.com/liyongkang123/Robust_LLM_Retriever_Eval.","url_abs":"https://arxiv.org/abs/2604.16576","url_pdf":"https://arxiv.org/pdf/2604.16576","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.16576","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.16576"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval","reach":null}],"summary":{"ran":1,"unverified":11},"by_repo_kind":{"found_in_text":{"samples":12,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b9edc5a66c170732","entry":"get_diag_of_cov","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"isotropy_utils/isoscore_numpy.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/isotropy_utils/isoscore_numpy.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9edc5a66c170732"}},{"code_sha256_prefix":"52a029392b311d5d","entry":"analyze_spectral_norms","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"draw_spectral_norm.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/draw_spectral_norm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52a029392b311d5d"}},{"code_sha256_prefix":"0076f5d1801f5224","entry":"get_attacked_queries_path","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"eval_attack.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/eval_attack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0076f5d1801f5224"}},{"code_sha256_prefix":"ca9d5c1e553d6cd2","entry":"load_attacked_queries","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"eval_attack.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/eval_attack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca9d5c1e553d6cd2"}},{"code_sha256_prefix":"f684968bb36ccc46","entry":"load_model_hf","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"utils/load_model.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/utils/load_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f684968bb36ccc46"}},{"code_sha256_prefix":"8b4fb7a187d33e21","entry":"normalize_diagonal","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"isotropy_utils/isoscore_numpy.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/isotropy_utils/isoscore_numpy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b4fb7a187d33e21"}},{"code_sha256_prefix":"4a8de46cfd435dbc","entry":"partition_score","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"isotropy_utils/existing_scores.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/isotropy_utils/existing_scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a8de46cfd435dbc"}},{"code_sha256_prefix":"5e0a2765ed1280c0","entry":"pca_normalization","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"isotropy_utils/isoscore_numpy.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/isotropy_utils/isoscore_numpy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e0a2765ed1280c0"}},{"code_sha256_prefix":"41b943b9b9ca9ff6","entry":"query_attack_main","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"eval_attack_data_statistics.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/eval_attack_data_statistics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41b943b9b9ca9ff6"}},{"code_sha256_prefix":"59073197852b3fff","entry":"remove_identical_ids","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"eval_attack.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/eval_attack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59073197852b3fff"}},{"code_sha256_prefix":"abf44238df547f42","entry":"save_attacked_queries","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"attack_queries_penha.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/attack_queries_penha.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"abf44238df547f42"}},{"code_sha256_prefix":"e3b26af1fc2ab62c","entry":"varex_score","repo":"liyongkang123/Robust_LLM_Retriever_Eval","repo_kind":"found_in_text","path":"isotropy_utils/existing_scores.py","file_url":"https://github.com/liyongkang123/Robust_LLM_Retriever_Eval/blob/HEAD/isotropy_utils/existing_scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3b26af1fc2ab62c"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}