{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rar-b-reasoning-as-retrieval-benchmark","title":"RAR-b: Reasoning as Retrieval Benchmark","arxiv_id":"2404.06347","date":"2024-04-09","proceeding":null,"authors":["Chenghao Xiao","G Thomas Hudson","Noura Al Moubayed"],"abstract":"Semantic textual similartiy (STS) and information retrieval tasks (IR) tasks have been the two major avenues to record the progress of embedding models in the past few years. Under the emerging Retrieval-augmented Generation (RAG) paradigm, we envision the need to evaluate next-level language understanding abilities of embedding models, and take a conscious look at the reasoning abilities stored in them. Addressing this, we pose the question: Can retrievers solve reasoning problems? By transforming reasoning tasks into retrieval tasks, we find that without specifically trained for reasoning-level language understanding, current state-of-the-art retriever models may still be far from being competent for playing the role of assisting LLMs, especially in reasoning-intensive tasks. Moreover, albeit trained to be aware of instructions, instruction-aware IR models are often better off without instructions in inference time for reasoning tasks, posing an overlooked retriever-LLM behavioral gap for the research community to align. However, recent decoder-based embedding models show great promise in narrowing the gap, highlighting the pathway for embedding models to achieve reasoning-level language understanding. We also show that, although current off-the-shelf re-ranker models fail on these tasks, injecting reasoning abilities into them through fine-tuning still appears easier than doing so to bi-encoders, and we are able to achieve state-of-the-art performance across all tasks by fine-tuning a reranking model. We release Reasoning as Retrieval Benchmark (RAR-b), a holistic suite of tasks and settings to evaluate the reasoning abilities stored in retriever models. RAR-b is available at https://github.com/gowitheflow-1998/RAR-b.","url_abs":"https://arxiv.org/abs/2404.06347v2","url_pdf":"https://arxiv.org/pdf/2404.06347v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rar-b-reasoning-as-retrieval-benchmark","repo_url":"https://github.com/gowitheflow-1998/rar-b","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"reranking","task_name":"Reranking"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":"sts","task_name":"STS"}],"methods":[{"method_slug":"aware","method_name":"AWARE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.06347","atlas_url":"https://app.syntology.ai/?focus=2404.06347","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.06347"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gowitheflow-1998/rar-b","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"eb9e8461fdee187e","entry":"capitalize_first","repo":"gowitheflow-1998/rar-b","repo_kind":"official","path":"rarb/dataset_utils.py","file_url":"https://github.com/gowitheflow-1998/rar-b/blob/HEAD/rarb/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb9e8461fdee187e"}},{"code_sha256_prefix":"e728d810a1e7e2c9","entry":"gritlm_instruction","repo":"gowitheflow-1998/rar-b","repo_kind":"official","path":"rarb/mcr_utils/model_utils.py","file_url":"https://github.com/gowitheflow-1998/rar-b/blob/HEAD/rarb/mcr_utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e728d810a1e7e2c9"}},{"code_sha256_prefix":"880b8d7cf30596ca","entry":"last_token_pool","repo":"gowitheflow-1998/rar-b","repo_kind":"official","path":"rarb/rarb_models/e5mistral.py","file_url":"https://github.com/gowitheflow-1998/rar-b/blob/HEAD/rarb/rarb_models/e5mistral.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"880b8d7cf30596ca"}},{"code_sha256_prefix":"aa489e8aefd0f91c","entry":"load_reasoning_raw","repo":"gowitheflow-1998/rar-b","repo_kind":"official","path":"rarb/dataset_utils.py","file_url":"https://github.com/gowitheflow-1998/rar-b/blob/HEAD/rarb/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa489e8aefd0f91c"}},{"code_sha256_prefix":"e49b2e3ed5800fc1","entry":"make_mcr_frame","repo":"gowitheflow-1998/rar-b","repo_kind":"official","path":"rarb/dataset_utils.py","file_url":"https://github.com/gowitheflow-1998/rar-b/blob/HEAD/rarb/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e49b2e3ed5800fc1"}},{"code_sha256_prefix":"2ded070fabe7bdc7","entry":"last_token_pool","repo":"gowitheflow-1998/rar-b","repo_kind":"official","path":"rarb/mcr_utils/model_utils.py","file_url":"https://github.com/gowitheflow-1998/rar-b/blob/HEAD/rarb/mcr_utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ded070fabe7bdc7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}