{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/alleviating-matthew-effect-of-offline","title":"Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation","arxiv_id":"2307.04571","date":"2023-07-10","proceeding":null,"authors":["Chongming Gao","Kexin Huang","Jiawei Chen","Yuan Zhang","Biao Li","Peng Jiang","Shiqi Wang","Zhong Zhang","Xiangnan He"],"abstract":"Offline reinforcement learning (RL), a technology that offline learns a policy from logged data without the need to interact with online environments, has become a favorable choice in decision-making processes like interactive recommendation. Offline RL faces the value overestimation problem. To address it, existing methods employ conservatism, e.g., by constraining the learned policy to be close to behavior policies or punishing the rarely visited state-action pairs. However, when applying such offline RL to recommendation, it will cause a severe Matthew effect, i.e., the rich get richer and the poor get poorer, by promoting popular items or categories while suppressing the less popular ones. It is a notorious issue that needs to be addressed in practical recommender systems. In this paper, we aim to alleviate the Matthew effect in offline RL-based recommendation. Through theoretical analyses, we find that the conservatism of existing methods fails in pursuing users' long-term satisfaction. It inspires us to add a penalty term to relax the pessimism on states with high entropy of the logging policy and indirectly penalizes actions leading to less diverse states. This leads to the main technical contribution of the work: Debiased model-based Offline RL (DORL) method. Experiments show that DORL not only captures user interests well but also alleviates the Matthew effect. The implementation is available via https://github.com/chongminggao/DORL-codes.","url_abs":"https://arxiv.org/abs/2307.04571v1","url_pdf":"https://arxiv.org/pdf/2307.04571v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"alleviating-matthew-effect-of-offline","repo_url":"https://github.com/chongminggao/dorl-codes","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"alleviating-matthew-effect-of-offline","repo_url":"https://github.com/ArronDZhang/ROLeR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"interactive-recommendation","task_name":"Interactive Recommendation"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.04571","atlas_url":"https://app.syntology.ai/?focus=2307.04571","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.04571"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ArronDZhang/ROLeR","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chongminggao/dorl-codes","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d1dad67b0318c176","entry":"get_args_dataset_specific","repo":"chongminggao/dorl-codes","repo_kind":"official","path":"run_worldModel_ensemble.py","file_url":"https://github.com/chongminggao/dorl-codes/blob/HEAD/run_worldModel_ensemble.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1dad67b0318c176"}},{"code_sha256_prefix":"3aab5f921e8397ba","entry":"load_mat_kuairand","repo":"ArronDZhang/ROLeR","repo_kind":"listed","path":"rew_reshape.py","file_url":"https://github.com/ArronDZhang/ROLeR/blob/HEAD/rew_reshape.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3aab5f921e8397ba"}},{"code_sha256_prefix":"f4afe07dc9c350bd","entry":"load_mat_kuairec","repo":"ArronDZhang/ROLeR","repo_kind":"listed","path":"rew_reshape.py","file_url":"https://github.com/ArronDZhang/ROLeR/blob/HEAD/rew_reshape.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4afe07dc9c350bd"}},{"code_sha256_prefix":"01e8a73f13b28644","entry":"loaddata","repo":"ArronDZhang/ROLeR","repo_kind":"listed","path":"results_for_paper/visual_main_figure_ROLeR.py","file_url":"https://github.com/ArronDZhang/ROLeR/blob/HEAD/results_for_paper/visual_main_figure_ROLeR.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01e8a73f13b28644"}},{"code_sha256_prefix":"e46acb9bd88e6bc4","entry":"loss_pointwise_Standard","repo":"chongminggao/dorl-codes","repo_kind":"official","path":"run_epsilongreedy.py","file_url":"https://github.com/chongminggao/dorl-codes/blob/HEAD/run_epsilongreedy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e46acb9bd88e6bc4"}},{"code_sha256_prefix":"53f2132cb23cf159","entry":"loss_pointwise_negative_Standard","repo":"chongminggao/dorl-codes","repo_kind":"official","path":"run_epsilongreedy.py","file_url":"https://github.com/chongminggao/dorl-codes/blob/HEAD/run_epsilongreedy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53f2132cb23cf159"}},{"code_sha256_prefix":"ae869ef797c1eab6","entry":"walk_paths","repo":"ArronDZhang/ROLeR","repo_kind":"listed","path":"results_for_paper/visual_main_figure_ROLeR.py","file_url":"https://github.com/ArronDZhang/ROLeR/blob/HEAD/results_for_paper/visual_main_figure_ROLeR.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae869ef797c1eab6"}},{"code_sha256_prefix":"2240d8f36743ff55","entry":"ew_min","repo":"ArronDZhang/ROLeR","repo_kind":"listed","path":"rew_reshape.py","file_url":"https://github.com/ArronDZhang/ROLeR/blob/HEAD/rew_reshape.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2240d8f36743ff55"}},{"code_sha256_prefix":"04883cfc46098d5f","entry":"organize_df","repo":"ArronDZhang/ROLeR","repo_kind":"listed","path":"results_for_paper/visual_main_figure_ROLeR.py","file_url":"https://github.com/ArronDZhang/ROLeR/blob/HEAD/results_for_paper/visual_main_figure_ROLeR.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04883cfc46098d5f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}