{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bordirlines-a-dataset-for-evaluating-cross","title":"BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented Generation","arxiv_id":"2410.01171","date":"2024-10-02","proceeding":null,"authors":["Bryan Li","Samar Haider","Fiona Luo","Adwait Agashe","Chris Callison-Burch"],"abstract":"Large language models excel at creative generation but continue to struggle with the issues of hallucination and bias. While retrieval-augmented generation (RAG) provides a framework for grounding LLMs' responses in accurate and up-to-date information, it still raises the question of bias: which sources should be selected for inclusion in the context? And how should their importance be weighted? In this paper, we study the challenge of cross-lingual RAG and present a dataset to investigate the robustness of existing systems at answering queries about geopolitical disputes, which exist at the intersection of linguistic, cultural, and political boundaries. Our dataset is sourced from Wikipedia pages containing information relevant to the given queries and we investigate the impact of including additional context, as well as the composition of this context in terms of language and source, on an LLM's response. Our results show that existing RAG systems continue to be challenged by cross-lingual use cases and suffer from a lack of consistency when they are provided with competing information in multiple languages. We present case studies to illustrate these issues and outline steps for future research to address these challenges. We make our dataset and code publicly available at https://github.com/manestay/bordIRlines.","url_abs":"https://arxiv.org/abs/2410.01171v1","url_pdf":"https://arxiv.org/pdf/2410.01171v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bordirlines-a-dataset-for-evaluating-cross","repo_url":"https://github.com/manestay/bordirlines","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.01171","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.01171"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/manestay/bordirlines","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":12,"unverified":4},"by_repo_kind":{"official":{"samples":16,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e0e86d5ea38ef05a","entry":"calc_ci","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/gen_all_results_table_multirun.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/gen_all_results_table_multirun.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e0e86d5ea38ef05a"}},{"code_sha256_prefix":"383171f09ff3e8c9","entry":"calculate_log_probabilities_for_candidates","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/generation.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"383171f09ff3e8c9"}},{"code_sha256_prefix":"37d3512123c2c4df","entry":"compare_dicts","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/check_query_json.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/check_query_json.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37d3512123c2c4df"}},{"code_sha256_prefix":"70b34c4796078f26","entry":"format_group","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/gen_all_results_table.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/gen_all_results_table.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70b34c4796078f26"}},{"code_sha256_prefix":"cfd231f817d2a9ad","entry":"gen_table","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/gen_all_results_table.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/gen_all_results_table.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfd231f817d2a9ad"}},{"code_sha256_prefix":"341667fa3efb145a","entry":"get_config","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/gen_all_results_table.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/gen_all_results_table.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"341667fa3efb145a"}},{"code_sha256_prefix":"ef76c6beda4c7143","entry":"get_territory_claimants_and_langs_map","repo":"manestay/bordirlines","repo_kind":"official","path":"lib.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/lib.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef76c6beda4c7143"}},{"code_sha256_prefix":"a7f3025a11ce10be","entry":"get_territory_langs_map","repo":"manestay/bordirlines","repo_kind":"official","path":"lib.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/lib.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7f3025a11ce10be"}},{"code_sha256_prefix":"9ea10d5ef8677130","entry":"get_title2id","repo":"manestay/bordirlines","repo_kind":"official","path":"get_wiki_articles.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/get_wiki_articles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ea10d5ef8677130"}},{"code_sha256_prefix":"ca72cfcf540903ab","entry":"load_dataset","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/generation_lib.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/generation_lib.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca72cfcf540903ab"}},{"code_sha256_prefix":"456ea162315edf33","entry":"load_json","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/check_query_json.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/check_query_json.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"456ea162315edf33"}},{"code_sha256_prefix":"317bb9bbcd22cdaf","entry":"load_with_progress","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/generation_lib.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/generation_lib.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"317bb9bbcd22cdaf"}},{"code_sha256_prefix":"d36bc8f80093fb9a","entry":"generate_next_tokens","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/generation.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d36bc8f80093fb9a"}},{"code_sha256_prefix":"b083d81036f81b00","entry":"get_wikipedia_langlinks","repo":"manestay/bordirlines","repo_kind":"official","path":"get_wiki_articles.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/get_wiki_articles.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b083d81036f81b00"}},{"code_sha256_prefix":"64e73289ea465067","entry":"load_borderlines_hf","repo":"manestay/bordirlines","repo_kind":"official","path":"lib.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/lib.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"64e73289ea465067"}},{"code_sha256_prefix":"b2d772cb7fde06e6","entry":"query_openai","repo":"manestay/bordirlines","repo_kind":"official","path":"generation/generation.py","file_url":"https://github.com/manestay/bordirlines/blob/HEAD/generation/generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2d772cb7fde06e6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}