{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-language-models-are-zero-shot-rankers","title":"Large Language Models are Zero-Shot Rankers for Recommender Systems","arxiv_id":"2305.08845","date":"2023-05-15","proceeding":null,"authors":["Yupeng Hou","Junjie Zhang","Zihan Lin","Hongyu Lu","Ruobing Xie","Julian McAuley","Wayne Xin Zhao"],"abstract":"Recently, large language models (LLMs) (e.g., GPT-4) have demonstrated impressive general-purpose task-solving abilities, including the potential to approach recommendation tasks. Along this line of research, this work aims to investigate the capacity of LLMs that act as the ranking model for recommender systems. We first formalize the recommendation problem as a conditional ranking task, considering sequential interaction histories as conditions and the items retrieved by other candidate generation models as candidates. To solve the ranking task by LLMs, we carefully design the prompting template and conduct extensive experiments on two widely-used datasets. We show that LLMs have promising zero-shot ranking abilities but (1) struggle to perceive the order of historical interactions, and (2) can be biased by popularity or item positions in the prompts. We demonstrate that these issues can be alleviated using specially designed prompting and bootstrapping strategies. Equipped with these insights, zero-shot LLMs can even challenge conventional recommendation models when ranking candidates are retrieved by multiple candidate generators. The code and processed datasets are available at https://github.com/RUCAIBox/LLMRank.","url_abs":"https://arxiv.org/abs/2305.08845v2","url_pdf":"https://arxiv.org/pdf/2305.08845v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"large-language-models-are-zero-shot-rankers","repo_url":"https://github.com/rucaibox/llmrank","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"large-language-models-are-zero-shot-rankers","repo_url":"https://github.com/rucaibox/tedrec","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.08845","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.08845"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rucaibox/tedrec","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rucaibox/llmrank","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dd88baa2e39b7ec5","entry":"get_user2count","repo":"rucaibox/llmrank","repo_kind":"official","path":"llmrank/data_process_amazon.py","file_url":"https://github.com/rucaibox/llmrank/blob/HEAD/llmrank/data_process_amazon.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd88baa2e39b7ec5"}},{"code_sha256_prefix":"423c5d0056286c09","entry":"load_meta_items","repo":"rucaibox/llmrank","repo_kind":"official","path":"llmrank/data_process_amazon.py","file_url":"https://github.com/rucaibox/llmrank/blob/HEAD/llmrank/data_process_amazon.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"423c5d0056286c09"}},{"code_sha256_prefix":"ffcd7499f55eb794","entry":"load_ratings","repo":"rucaibox/llmrank","repo_kind":"official","path":"llmrank/data_process_amazon.py","file_url":"https://github.com/rucaibox/llmrank/blob/HEAD/llmrank/data_process_amazon.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffcd7499f55eb794"}},{"code_sha256_prefix":"38d0b46943ff2ffc","entry":"load_single_source","repo":"rucaibox/llmrank","repo_kind":"official","path":"llmrank/mix_sources.py","file_url":"https://github.com/rucaibox/llmrank/blob/HEAD/llmrank/mix_sources.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38d0b46943ff2ffc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}