{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-the-evaluation-for-conversational","title":"Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models","arxiv_id":"2305.13112","date":"2023-05-22","proceeding":null,"authors":["Xiaolei Wang","Xinyu Tang","Wayne Xin Zhao","Jingyuan Wang","Ji-Rong Wen"],"abstract":"The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs), which rely on natural language conversations to satisfy user needs. In this paper, we embark on an investigation into the utilization of ChatGPT for conversational recommendation, revealing the inadequacy of the existing evaluation protocol. It might over-emphasize the matching with the ground-truth items or utterances generated by human annotators, while neglecting the interactive nature of being a capable CRS. To overcome the limitation, we further propose an interactive Evaluation approach based on LLMs named iEvaLM that harnesses LLM-based user simulators. Our evaluation approach can simulate various interaction scenarios between users and systems. Through the experiments on two publicly available CRS datasets, we demonstrate notable improvements compared to the prevailing evaluation protocol. Furthermore, we emphasize the evaluation of explainability, and ChatGPT showcases persuasive explanation generation for its recommendations. Our study contributes to a deeper comprehension of the untapped potential of LLMs for CRSs and provides a more flexible and easy-to-use evaluation framework for future research endeavors. The codes and data are publicly available at https://github.com/RUCAIBox/iEvaLM-CRS.","url_abs":"https://arxiv.org/abs/2305.13112v2","url_pdf":"https://arxiv.org/pdf/2305.13112v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-the-evaluation-for-conversational","repo_url":"https://github.com/rucaibox/ievalm-crs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"conversational-recommendation","task_name":"Conversational Recommendation"},{"task_slug":"explanation-generation","task_name":"Explanation Generation"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.13112","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.13112"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rucaibox/ievalm-crs","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/RUCAIBox/iEvaLM-CRS","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":1,"unverified":15},"by_repo_kind":{"official":{"samples":17,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1052ac20f3dfe2dc","entry":"annotate","repo":"RUCAIBox/iEvaLM-CRS","repo_kind":"official","path":"src/model/CHATGPT.py","file_url":"https://github.com/RUCAIBox/iEvaLM-CRS/blob/HEAD/src/model/CHATGPT.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1052ac20f3dfe2dc"}},{"code_sha256_prefix":"2419551feec87f10","entry":"my_wait_exponential","repo":"RUCAIBox/iEvaLM-CRS","repo_kind":"official","path":"src/model/CHATGPT.py","file_url":"https://github.com/RUCAIBox/iEvaLM-CRS/blob/HEAD/src/model/CHATGPT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2419551feec87f10"}},{"code_sha256_prefix":"7fab3666beecf259","entry":"CHATGPT","repo":"RUCAIBox/iEvaLM-CRS","repo_kind":"official","path":"src/model/CHATGPT.py","file_url":"https://github.com/RUCAIBox/iEvaLM-CRS/blob/HEAD/src/model/CHATGPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fab3666beecf259"}},{"code_sha256_prefix":"4b4ed6d15480f5a5","entry":"annotate","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"script/cache_item.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/script/cache_item.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b4ed6d15480f5a5"}},{"code_sha256_prefix":"285d00ffcd75ae67","entry":"annotate","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"src/model/CHATGPT.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/src/model/CHATGPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"285d00ffcd75ae67"}},{"code_sha256_prefix":"e92d3fa7cbc2a58c","entry":"annotate_chat","repo":"RUCAIBox/iEvaLM-CRS","repo_kind":"official","path":"src/model/CHATGPT.py","file_url":"https://github.com/RUCAIBox/iEvaLM-CRS/blob/HEAD/src/model/CHATGPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e92d3fa7cbc2a58c"}},{"code_sha256_prefix":"a3d50124d306d006","entry":"annotate_chat","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"src/model/CHATGPT.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/src/model/CHATGPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3d50124d306d006"}},{"code_sha256_prefix":"bcfca1295d83a6e4","entry":"annotate_completion","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"script/ask.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/script/ask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bcfca1295d83a6e4"}},{"code_sha256_prefix":"c0900257616e5ccb","entry":"get_instruction","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"script/ask.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/script/ask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0900257616e5ccb"}},{"code_sha256_prefix":"8cfed84e6fae7269","entry":"get_instruction","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"script/chat.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/script/chat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cfed84e6fae7269"}},{"code_sha256_prefix":"ed9662882a67bfc4","entry":"get_model_args","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"script/ask.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/script/ask.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed9662882a67bfc4"}},{"code_sha256_prefix":"582ed53a923214b4","entry":"get_model_args","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"script/chat.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/script/chat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"582ed53a923214b4"}},{"code_sha256_prefix":"4fe63c6cb46bf907","entry":"load_jsonl_data","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"src/model/utils.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/src/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fe63c6cb46bf907"}},{"code_sha256_prefix":"fcd2cd328d39b785","entry":"my_before_sleep","repo":"RUCAIBox/iEvaLM-CRS","repo_kind":"official","path":"src/model/CHATGPT.py","file_url":"https://github.com/RUCAIBox/iEvaLM-CRS/blob/HEAD/src/model/CHATGPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcd2cd328d39b785"}},{"code_sha256_prefix":"29efbd3fe0422744","entry":"my_stop_after_attempt","repo":"RUCAIBox/iEvaLM-CRS","repo_kind":"official","path":"src/model/CHATGPT.py","file_url":"https://github.com/RUCAIBox/iEvaLM-CRS/blob/HEAD/src/model/CHATGPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29efbd3fe0422744"}},{"code_sha256_prefix":"82d4d951d061c46e","entry":"sample_data","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"src/model/utils.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/src/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82d4d951d061c46e"}},{"code_sha256_prefix":"f1a7898875d9e106","entry":"simple_collate","repo":"rucaibox/ievalm-crs","repo_kind":"official","path":"src/model/utils.py","file_url":"https://github.com/rucaibox/ievalm-crs/blob/HEAD/src/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1a7898875d9e106"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}