{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prefrec-preference-based-recommender-systems","title":"PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement","arxiv_id":"2212.02779","date":"2022-12-06","proceeding":null,"authors":["Wanqi Xue","Qingpeng Cai","Zhenghai Xue","Shuo Sun","Shuchang Liu","Dong Zheng","Peng Jiang","Kun Gai","Bo An"],"abstract":"Current advances in recommender systems have been remarkably successful in optimizing immediate engagement. However, long-term user engagement, a more desirable performance metric, remains difficult to improve. Meanwhile, recent reinforcement learning (RL) algorithms have shown their effectiveness in a variety of long-term goal optimization tasks. For this reason, RL is widely considered as a promising framework for optimizing long-term user engagement in recommendation. Though promising, the application of RL heavily relies on well-designed rewards, but designing rewards related to long-term user engagement is quite difficult. To mitigate the problem, we propose a novel paradigm, recommender systems with human preferences (or Preference-based Recommender systems), which allows RL recommender systems to learn from preferences about users historical behaviors rather than explicitly defined rewards. Such preferences are easily accessible through techniques such as crowdsourcing, as they do not require any expert knowledge. With PrefRec, we can fully exploit the advantages of RL in optimizing long-term goals, while avoiding complex reward engineering. PrefRec uses the preferences to automatically train a reward function in an end-to-end manner. The reward function is then used to generate learning signals to train the recommendation policy. Furthermore, we design an effective optimization method for PrefRec, which uses an additional value function, expectile regression and reward model pre-training to improve the performance. We conduct experiments on a variety of long-term user engagement optimization tasks. The results show that PrefRec significantly outperforms previous state-of-the-art methods in all the tasks.","url_abs":"https://arxiv.org/abs/2212.02779v2","url_pdf":"https://arxiv.org/pdf/2212.02779v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prefrec-preference-based-recommender-systems","repo_url":"https://github.com/chongminggao/easyrl4rec","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.02779","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.02779"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chongminggao/easyrl4rec","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"listed":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2fb349ac030ba393","entry":"build_input_features","repo":"chongminggao/easyrl4rec","repo_kind":"listed","path":"src/DeepCTR-Torch/deepctr_torch/inputs.py","file_url":"https://github.com/chongminggao/easyrl4rec/blob/HEAD/src/DeepCTR-Torch/deepctr_torch/inputs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fb349ac030ba393"}},{"code_sha256_prefix":"2aea84481b6e2f65","entry":"combined_dnn_input","repo":"chongminggao/easyrl4rec","repo_kind":"listed","path":"src/DeepCTR-Torch/deepctr_torch/inputs.py","file_url":"https://github.com/chongminggao/easyrl4rec/blob/HEAD/src/DeepCTR-Torch/deepctr_torch/inputs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2aea84481b6e2f65"}},{"code_sha256_prefix":"47c8d841197bf979","entry":"compute_exposure_effect","repo":"chongminggao/easyrl4rec","repo_kind":"listed","path":"src/core/envs/BaseData.py","file_url":"https://github.com/chongminggao/easyrl4rec/blob/HEAD/src/core/envs/BaseData.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47c8d841197bf979"}},{"code_sha256_prefix":"79a03988c89913ac","entry":"get_distance_mat","repo":"chongminggao/easyrl4rec","repo_kind":"listed","path":"src/core/envs/BaseData.py","file_url":"https://github.com/chongminggao/easyrl4rec/blob/HEAD/src/core/envs/BaseData.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79a03988c89913ac"}},{"code_sha256_prefix":"ba1c9d87dd13207a","entry":"get_feature_names","repo":"chongminggao/easyrl4rec","repo_kind":"listed","path":"src/DeepCTR-Torch/deepctr_torch/inputs.py","file_url":"https://github.com/chongminggao/easyrl4rec/blob/HEAD/src/DeepCTR-Torch/deepctr_torch/inputs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba1c9d87dd13207a"}},{"code_sha256_prefix":"f307d3ef6b470b82","entry":"load_dfs","repo":"chongminggao/easyrl4rec","repo_kind":"listed","path":"visual_results/results_utils.py","file_url":"https://github.com/chongminggao/easyrl4rec/blob/HEAD/visual_results/results_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f307d3ef6b470b82"}},{"code_sha256_prefix":"a9be072f14f5e318","entry":"loaddata","repo":"chongminggao/easyrl4rec","repo_kind":"listed","path":"visual_results/results_utils.py","file_url":"https://github.com/chongminggao/easyrl4rec/blob/HEAD/visual_results/results_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9be072f14f5e318"}},{"code_sha256_prefix":"4ccf05078d13f3e3","entry":"test_taobao","repo":"chongminggao/easyrl4rec","repo_kind":"listed","path":"src/core/evaluation/evaluator_static.py","file_url":"https://github.com/chongminggao/easyrl4rec/blob/HEAD/src/core/evaluation/evaluator_static.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ccf05078d13f3e3"}},{"code_sha256_prefix":"5bec6bc8fc3f8282","entry":"walk_paths","repo":"chongminggao/easyrl4rec","repo_kind":"listed","path":"visual_results/results_utils.py","file_url":"https://github.com/chongminggao/easyrl4rec/blob/HEAD/visual_results/results_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bec6bc8fc3f8282"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}