{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-difficulty-of-evaluating-baselines-a","title":"On the Difficulty of Evaluating Baselines: A Study on Recommender Systems","arxiv_id":"1905.01395","date":"2019-05-04","proceeding":null,"authors":["Steffen Rendle","Li Zhang","Yehuda Koren"],"abstract":"Numerical evaluations with comparisons to baselines play a central role when judging research in recommender systems. In this paper, we show that running baselines properly is difficult. We demonstrate this issue on two extensively studied datasets. First, we show that results for baselines that have been used in numerous publications over the past five years for the Movielens 10M benchmark are suboptimal. With a careful setup of a vanilla matrix factorization baseline, we are not only able to improve upon the reported results for this baseline but even outperform the reported results of any newly proposed method. Secondly, we recap the tremendous effort that was required by the community to obtain high quality results for simple methods on the Netflix Prize. Our results indicate that empirical findings in research papers are questionable unless they were obtained on standardized benchmarks where baselines have been tuned extensively by the research community.","url_abs":"https://arxiv.org/abs/1905.01395v1","url_pdf":"https://arxiv.org/pdf/1905.01395v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-difficulty-of-evaluating-baselines-a","repo_url":"https://github.com/srendle/libfm","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"on-the-difficulty-of-evaluating-baselines-a","repo_url":"https://github.com/tohtsky/myFM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"collaborative-filtering","task_name":"Collaborative Filtering"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/collaborative-filtering-on-movielens-10m","task":"Recommendation Systems","dataset":"MovieLens 10M","model":"Bayesian timeSVD++ flipped","rank_in_archive_order":1,"of":17,"metrics":{"RMSE":"0.7485"},"uses_additional_data":false},{"leaderboard":"/sota/collaborative-filtering-on-movielens-10m","task":"Recommendation Systems","dataset":"MovieLens 10M","model":"Bayesian timeSVD++","rank_in_archive_order":2,"of":17,"metrics":{"RMSE":"0.7523"},"uses_additional_data":false},{"leaderboard":"/sota/collaborative-filtering-on-movielens-10m","task":"Recommendation Systems","dataset":"MovieLens 10M","model":"Bayesian SVD++","rank_in_archive_order":3,"of":17,"metrics":{"RMSE":"0.7563"},"uses_additional_data":false},{"leaderboard":"/sota/collaborative-filtering-on-movielens-10m","task":"Recommendation Systems","dataset":"MovieLens 10M","model":"SGD MF","rank_in_archive_order":7,"of":17,"metrics":{"RMSE":"0.772"},"uses_additional_data":false},{"leaderboard":"/sota/collaborative-filtering-on-movielens-10m","task":"Recommendation Systems","dataset":"MovieLens 10M","model":"U-RBM","rank_in_archive_order":15,"of":17,"metrics":{"RMSE":"0.823"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1905.01395","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1905.01395"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/srendle/libfm","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tohtsky/myFM","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"792f1246b7c00a20","entry":"gen_dummy_rating_df","repo":"tohtsky/myFM","repo_kind":"listed","path":"src/myfm/utils/dummy_data.py","file_url":"https://github.com/tohtsky/myFM/blob/HEAD/src/myfm/utils/dummy_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"792f1246b7c00a20"}},{"code_sha256_prefix":"a8b1400cbff6b403","entry":"read_ml1m10m_df","repo":"tohtsky/myFM","repo_kind":"listed","path":"src/myfm/utils/benchmark_data/movielens1M_data.py","file_url":"https://github.com/tohtsky/myFM/blob/HEAD/src/myfm/utils/benchmark_data/movielens1M_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8b1400cbff6b403"}},{"code_sha256_prefix":"a09658e544e71876","entry":"train_test_split_with_kfold","repo":"tohtsky/myFM","repo_kind":"listed","path":"src/myfm/utils/benchmark_data/loader_base.py","file_url":"https://github.com/tohtsky/myFM/blob/HEAD/src/myfm/utils/benchmark_data/loader_base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a09658e544e71876"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}