{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/widespread-flaws-in-offline-evaluation-of","title":"Widespread Flaws in Offline Evaluation of Recommender Systems","arxiv_id":"2307.14951","date":"2023-07-27","proceeding":null,"authors":["Balázs Hidasi","Ádám Tibor Czapp"],"abstract":"Even though offline evaluation is just an imperfect proxy of online performance -- due to the interactive nature of recommenders -- it will probably remain the primary way of evaluation in recommender systems research for the foreseeable future, since the proprietary nature of production recommenders prevents independent validation of A/B test setups and verification of online results. Therefore, it is imperative that offline evaluation setups are as realistic and as flawless as they can be. Unfortunately, evaluation flaws are quite common in recommender systems research nowadays, due to later works copying flawed evaluation setups from their predecessors without questioning their validity. In the hope of improving the quality of offline evaluation of recommender systems, we discuss four of these widespread flaws and why researchers should avoid them.","url_abs":"https://arxiv.org/abs/2307.14951v1","url_pdf":"https://arxiv.org/pdf/2307.14951v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"widespread-flaws-in-offline-evaluation-of","repo_url":"https://github.com/hidasib/GRU4Rec","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"widespread-flaws-in-offline-evaluation-of","repo_url":"https://github.com/hidasib/recsys_eval_flaws","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"widespread-flaws-in-offline-evaluation-of","repo_url":"https://github.com/hidasib/gru4rec_pytorch_official","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"widespread-flaws-in-offline-evaluation-of","repo_url":"https://github.com/hidasib/gru4rec_tensorflow_official","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.14951","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.14951"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hidasib/recsys_eval_flaws","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hidasib/GRU4Rec","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hidasib/gru4rec_pytorch_official","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hidasib/gru4rec_tensorflow_official","reach":{"status":"ok"}}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"cf315d2b289d3a7a","entry":"compute_offset","repo":"hidasib/GRU4Rec","repo_kind":"official","path":"datatools.py","file_url":"https://github.com/hidasib/GRU4Rec/blob/HEAD/datatools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cf315d2b289d3a7a"}},{"code_sha256_prefix":"751f0bb03ae1713a","entry":"recall_mrr_from_ranks","repo":"hidasib/recsys_eval_flaws","repo_kind":"official","path":"sampling/inference_sampling.py","file_url":"https://github.com/hidasib/recsys_eval_flaws/blob/HEAD/sampling/inference_sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"751f0bb03ae1713a"}},{"code_sha256_prefix":"92f4d857f35d1afd","entry":"evaluate_sessions","repo":"hidasib/GRU4Rec","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/hidasib/GRU4Rec/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"92f4d857f35d1afd"}},{"code_sha256_prefix":"fa46b6b875d49557","entry":"evaluate_sessions_batch","repo":"hidasib/GRU4Rec","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/hidasib/GRU4Rec/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fa46b6b875d49557"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}