{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/first-faster-improved-listwise-reranking-with","title":"FIRST: Faster Improved Listwise Reranking with Single Token Decoding","arxiv_id":"2406.15657","date":"2024-06-21","proceeding":null,"authors":["Revanth Gangi Reddy","JaeHyeok Doo","Yifei Xu","Md Arafat Sultan","Deevya Swain","Avirup Sil","Heng Ji"],"abstract":"Large Language Models (LLMs) have significantly advanced the field of information retrieval, particularly for reranking. Listwise LLM rerankers have showcased superior performance and generalizability compared to existing supervised approaches. However, conventional listwise LLM reranking methods lack efficiency as they provide ranking output in the form of a generated ordered sequence of candidate passage identifiers. Further, they are trained with the typical language modeling objective, which treats all ranking errors uniformly--potentially at the cost of misranking highly relevant passages. Addressing these limitations, we introduce FIRST, a novel listwise LLM reranking approach leveraging the output logits of the first generated identifier to directly obtain a ranked ordering of the candidates. Further, we incorporate a learning-to-rank loss during training, prioritizing ranking accuracy for the more relevant passages. Empirical results demonstrate that FIRST accelerates inference by 50% while maintaining a robust ranking performance with gains across the BEIR benchmark. Finally, to illustrate the practical effectiveness of listwise LLM rerankers, we investigate their application in providing relevance feedback for retrievers during inference. Our results show that LLM rerankers can provide a stronger distillation signal compared to cross-encoders, yielding substantial improvements in retriever recall after relevance feedback.","url_abs":"https://arxiv.org/abs/2406.15657v1","url_pdf":"https://arxiv.org/pdf/2406.15657v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"first-faster-improved-listwise-reranking-with","repo_url":"https://github.com/gangiswag/llm-reranker","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"learning-to-rank","task_name":"Learning-To-Rank"},{"task_slug":"reranking","task_name":"Reranking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2406.15657","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.15657"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gangiswag/llm-reranker","reach":{"status":"ok"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"9adb9c459cc4c8f0","entry":"combine_faiss_results","repo":"gangiswag/llm-reranker","repo_kind":"official","path":"tevatron/src/tevatron/faiss_retriever/reducer.py","file_url":"https://github.com/gangiswag/llm-reranker/blob/HEAD/tevatron/src/tevatron/faiss_retriever/reducer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9adb9c459cc4c8f0"}},{"code_sha256_prefix":"fd973ee45608bfd5","entry":"get_dense_rep","repo":"gangiswag/llm-reranker","repo_kind":"official","path":"tevatron/src/tevatron/trainer.py","file_url":"https://github.com/gangiswag/llm-reranker/blob/HEAD/tevatron/src/tevatron/trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd973ee45608bfd5"}},{"code_sha256_prefix":"4d0f740bd3b8e05f","entry":"pickle_load","repo":"gangiswag/llm-reranker","repo_kind":"official","path":"tevatron/src/tevatron/faiss_retriever/__main__.py","file_url":"https://github.com/gangiswag/llm-reranker/blob/HEAD/tevatron/src/tevatron/faiss_retriever/__main__.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d0f740bd3b8e05f"}},{"code_sha256_prefix":"ab186523204316c5","entry":"search_queries","repo":"gangiswag/llm-reranker","repo_kind":"official","path":"tevatron/src/tevatron/faiss_retriever/__main__.py","file_url":"https://github.com/gangiswag/llm-reranker/blob/HEAD/tevatron/src/tevatron/faiss_retriever/__main__.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab186523204316c5"}},{"code_sha256_prefix":"a68ca4e05b568068","entry":"split_dense_inputs","repo":"gangiswag/llm-reranker","repo_kind":"official","path":"tevatron/src/tevatron/trainer.py","file_url":"https://github.com/gangiswag/llm-reranker/blob/HEAD/tevatron/src/tevatron/trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a68ca4e05b568068"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}