{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-generalizability-and-predictability-of","title":"On the Generalizability and Predictability of Recommender Systems","arxiv_id":"2206.11886","date":"2022-06-23","proceeding":null,"authors":["Duncan McElfresh","Sujay Khandagale","Jonathan Valverde","John P. Dickerson","Colin White"],"abstract":"While other areas of machine learning have seen more and more automation, designing a high-performing recommender system still requires a high level of human effort. Furthermore, recent work has shown that modern recommender system algorithms do not always improve over well-tuned baselines. A natural follow-up question is, \"how do we choose the right algorithm for a new dataset and performance metric?\" In this work, we start by giving the first large-scale study of recommender system approaches by comparing 18 algorithms and 100 sets of hyperparameters across 85 datasets and 315 metrics. We find that the best algorithms and hyperparameters are highly dependent on the dataset and performance metric, however, there are also strong correlations between the performance of each algorithm and various meta-features of the datasets. Motivated by these findings, we create RecZilla, a meta-learning approach to recommender systems that uses a model to predict the best algorithm and hyperparameters for new, unseen datasets. By using far more meta-training data than prior work, RecZilla is able to substantially reduce the level of human involvement when faced with a new recommender system application. We not only release our code and pretrained RecZilla models, but also all of our raw experimental results, so that practitioners can train a RecZilla model for their desired performance metric: https://github.com/naszilla/reczilla.","url_abs":"https://arxiv.org/abs/2206.11886v2","url_pdf":"https://arxiv.org/pdf/2206.11886v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-generalizability-and-predictability-of","repo_url":"https://github.com/naszilla/reczilla","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"meta-learning","task_name":"Meta-Learning"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.11886","atlas_url":"https://app.syntology.ai/?focus=2206.11886","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.11886"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/naszilla/reczilla","reach":null}],"summary":{"ran_honours":3,"ran_draft_wrong":4,"ran_violates":1,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":13,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"203f2961b800f53b","entry":"compute_feature_corrs","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"203f2961b800f53b"}},{"code_sha256_prefix":"f95d2eb37f3c4bdd","entry":"dataset_family_lookup","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f95d2eb37f3c4bdd"}},{"code_sha256_prefix":"404f06fb48571615","entry":"filter_for_cunha","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"404f06fb48571615"}},{"code_sha256_prefix":"36ab817c64f4674d","entry":"get_metafeats","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"36ab817c64f4674d"}},{"code_sha256_prefix":"69d8d93b52b4825f","entry":"is_time_metric","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69d8d93b52b4825f"}},{"code_sha256_prefix":"2aea19c0e6e97c49","entry":"run_metalearner","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2aea19c0e6e97c49"}},{"code_sha256_prefix":"25df574617c68a5b","entry":"select_features","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25df574617c68a5b"}},{"code_sha256_prefix":"e77042e64036e265","entry":"w_cov","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e77042e64036e265"}},{"code_sha256_prefix":"affdef6e712df04e","entry":"weighted_corr","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"affdef6e712df04e"}},{"code_sha256_prefix":"520f5c5a2bc9a05c","entry":"alg_feature_selection_featurized","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"520f5c5a2bc9a05c"}},{"code_sha256_prefix":"449d6c338ab3dd80","entry":"my_print","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"449d6c338ab3dd80"}},{"code_sha256_prefix":"0dcb984da98265e6","entry":"reczilla_train","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0dcb984da98265e6"}},{"code_sha256_prefix":"f7c30d0cf631977f","entry":"select_algs","repo":"naszilla/reczilla","repo_kind":"official","path":"RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","file_url":"https://github.com/naszilla/reczilla/blob/HEAD/RecSys2019_DeepLearning_Evaluation/ReczillaClassifier/run_reczilla.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f7c30d0cf631977f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}