{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llmrec-benchmarking-large-language-models-on","title":"LLMRec: Benchmarking Large Language Models on Recommendation Task","arxiv_id":"2308.12241","date":"2023-08-23","proceeding":null,"authors":["Junling Liu","Chao Liu","Peilin Zhou","Qichen Ye","Dading Chong","Kang Zhou","Yueqi Xie","Yuwei Cao","Shoujin Wang","Chenyu You","Philip S. Yu"],"abstract":"Recently, the fast development of Large Language Models (LLMs) such as ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of conversational models. However, the application of LLMs in the recommendation domain has not been thoroughly investigated. To bridge this gap, we propose LLMRec, a LLM-based recommender system designed for benchmarking LLMs on various recommendation tasks. Specifically, we benchmark several popular off-the-shelf LLMs, such as ChatGPT, LLaMA, ChatGLM, on five recommendation tasks, including rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization. Furthermore, we investigate the effectiveness of supervised finetuning to improve LLMs' instruction compliance ability. The benchmark results indicate that LLMs displayed only moderate proficiency in accuracy-based tasks such as sequential and direct recommendation. However, they demonstrated comparable performance to state-of-the-art methods in explainability-based tasks. We also conduct qualitative evaluations to further evaluate the quality of contents generated by different models, and the results show that LLMs can truly understand the provided information and generate clearer and more reasonable results. We aspire that this benchmark will serve as an inspiration for researchers to delve deeper into the potential of LLMs in enhancing recommendation performance. Our codes, processed data and benchmark results are available at https://github.com/williamliujl/LLMRec.","url_abs":"https://arxiv.org/abs/2308.12241v1","url_pdf":"https://arxiv.org/pdf/2308.12241v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llmrec-benchmarking-large-language-models-on","repo_url":"https://github.com/williamliujl/llmrec","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"explanation-generation","task_name":"Explanation Generation"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"},{"task_slug":"sequential-recommendation","task_name":"Sequential Recommendation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.12241","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.12241"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/williamliujl/llmrec","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"unverified":3},"by_repo_kind":{"official":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"49289a7f94c227bd","entry":"compute_bleu","repo":"williamliujl/llmrec","repo_kind":"official","path":"utils/evaluate/bleu.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/utils/evaluate/bleu.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"49289a7f94c227bd"}},{"code_sha256_prefix":"c66438d5f4a93e80","entry":"evaluate_old","repo":"williamliujl/llmrec","repo_kind":"official","path":"utils/evaluate/metrics4rec.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/utils/evaluate/metrics4rec.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c66438d5f4a93e80"}},{"code_sha256_prefix":"2fa27f253fa37d79","entry":"hit_at_k","repo":"williamliujl/llmrec","repo_kind":"official","path":"utils/evaluate/metrics4rec.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/utils/evaluate/metrics4rec.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2fa27f253fa37d79"}},{"code_sha256_prefix":"cb6e8f7a5b600f81","entry":"load_instruction","repo":"williamliujl/llmrec","repo_kind":"official","path":"infer_rec.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/infer_rec.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb6e8f7a5b600f81"}},{"code_sha256_prefix":"74ce630ade2232fd","entry":"load_instruction_from_csv","repo":"williamliujl/llmrec","repo_kind":"official","path":"infer_rec.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/infer_rec.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"74ce630ade2232fd"}},{"code_sha256_prefix":"86965fc714650816","entry":"recall_at_k","repo":"williamliujl/llmrec","repo_kind":"official","path":"utils/evaluate/metrics4rec.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/utils/evaluate/metrics4rec.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86965fc714650816"}},{"code_sha256_prefix":"815c9f02431395eb","entry":"rouge_l_sentence_level","repo":"williamliujl/llmrec","repo_kind":"official","path":"utils/evaluate/rouge.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/utils/evaluate/rouge.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"815c9f02431395eb"}},{"code_sha256_prefix":"eb37c01f4067f842","entry":"rouge_n","repo":"williamliujl/llmrec","repo_kind":"official","path":"utils/evaluate/rouge.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/utils/evaluate/rouge.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb37c01f4067f842"}},{"code_sha256_prefix":"b8d4a80a37d157bc","entry":"two_seq_same","repo":"williamliujl/llmrec","repo_kind":"official","path":"utils/evaluate/utils.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/utils/evaluate/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8d4a80a37d157bc"}},{"code_sha256_prefix":"c212cb768b5261af","entry":"bleu_score","repo":"williamliujl/llmrec","repo_kind":"official","path":"utils/evaluate/utils.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/utils/evaluate/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c212cb768b5261af"}},{"code_sha256_prefix":"67d8d100e11a6652","entry":"rouge_l_summary_level","repo":"williamliujl/llmrec","repo_kind":"official","path":"utils/evaluate/rouge.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/utils/evaluate/rouge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67d8d100e11a6652"}},{"code_sha256_prefix":"e173835ad8f5c718","entry":"rouge_score","repo":"williamliujl/llmrec","repo_kind":"official","path":"utils/evaluate/utils.py","file_url":"https://github.com/williamliujl/llmrec/blob/HEAD/utils/evaluate/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e173835ad8f5c718"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}