{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-01997","title":"Exploring Diversity, Novelty, and Popularity Bias in ChatGPT's Recommendations","arxiv_id":"2601.01997","date":"2026-01-05","proceeding":null,"authors":["Dario Di Palma","Giovanni Maria Biancofiore","Vito Walter Anelli","Fedelucio Narducci","Tommaso Di Noia"],"abstract":"ChatGPT has emerged as a versatile tool, demonstrating capabilities across diverse domains. Given these successes, the Recommender Systems (RSs) community has begun investigating its applications within recommendation scenarios primarily focusing on accuracy. While the integration of ChatGPT into RSs has garnered significant attention, a comprehensive analysis of its performance across various dimensions remains largely unexplored. Specifically, the capabilities of providing diverse and novel recommendations or exploring potential biases such as popularity bias have not been thoroughly examined. As the use of these models continues to expand, understanding these aspects is crucial for enhancing user satisfaction and achieving long-term personalization. This study investigates the recommendations provided by ChatGPT-3.5 and ChatGPT-4 by assessing ChatGPT's capabilities in terms of diversity, novelty, and popularity bias. We evaluate these models on three distinct datasets and assess their performance in Top-N recommendation and cold-start scenarios. The findings reveal that ChatGPT-4 matches or surpasses traditional recommenders, demonstrating the ability to balance novelty and diversity in recommendations. Furthermore, in the cold-start scenario, ChatGPT models exhibit superior performance in both accuracy and novelty, suggesting they can be particularly beneficial for new users. This research highlights the strengths and limitations of ChatGPT's recommendations, offering new perspectives on the capacity of these models to provide recommendations beyond accuracy-focused metrics.","url_abs":"https://arxiv.org/abs/2601.01997","url_pdf":"https://arxiv.org/pdf/2601.01997","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2601.01997","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.01997"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":13},"by_repo_kind":{"found_in_text":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"503b970cb01911b8","entry":"calculate_jaccard_similarity","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/evaluation/evaluation_metrics.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/evaluation/evaluation_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"503b970cb01911b8"}},{"code_sha256_prefix":"9f7b19d3800ee846","entry":"check_item_similarity_mistral_7b","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/utils/llm_utils.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/utils/llm_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f7b19d3800ee846"}},{"code_sha256_prefix":"60032eaa52f13e20","entry":"check_item_similarity_mistral_7b_v2","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/utils/llm_utils.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/utils/llm_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60032eaa52f13e20"}},{"code_sha256_prefix":"201c1841b5acd99a","entry":"compute_time","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/general_main.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/general_main.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"201c1841b5acd99a"}},{"code_sha256_prefix":"95c969e396783a94","entry":"list_tsv_files","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/run_elliot.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/run_elliot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"95c969e396783a94"}},{"code_sha256_prefix":"f672ef26c322bb6d","entry":"load_attribute_file","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/utils/facebook_book_utils.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/utils/facebook_book_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f672ef26c322bb6d"}},{"code_sha256_prefix":"730851a560fe9bbc","entry":"load_feature_names","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/utils/facebook_book_utils.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/utils/facebook_book_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"730851a560fe9bbc"}},{"code_sha256_prefix":"d14be71225d00490","entry":"minmax_scaler_for_users","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/utils/hetrec2011_utils.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/utils/hetrec2011_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d14be71225d00490"}},{"code_sha256_prefix":"395edfaf5a1e63d4","entry":"modify_tsv","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/run_elliot.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/run_elliot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"395edfaf5a1e63d4"}},{"code_sha256_prefix":"87e8fd098c99dcde","entry":"prompt_gen","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/utils/llm_utils.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/utils/llm_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"87e8fd098c99dcde"}},{"code_sha256_prefix":"83d405c8b9fd8b14","entry":"search_50_unlisten_artist","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/utils/load_userknn.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/utils/load_userknn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83d405c8b9fd8b14"}},{"code_sha256_prefix":"ab8604da3ff45feb","entry":"search_50_unread_book","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/utils/load_userknn.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/utils/load_userknn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab8604da3ff45feb"}},{"code_sha256_prefix":"1809ffc4cbec057e","entry":"search_50_unseen_movies","repo":"sisinflab/beyond-accuracy-recsys-chatgpt","repo_kind":"found_in_text","path":"code/utils/load_userknn.py","file_url":"https://github.com/sisinflab/beyond-accuracy-recsys-chatgpt/blob/HEAD/code/utils/load_userknn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1809ffc4cbec057e"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}