{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/agentsociety-challenge-designing-llm-agents","title":"AgentSociety Challenge: Designing LLM Agents for User Modeling and Recommendation on Web Platforms","arxiv_id":"2502.18754","date":"2025-02-26","proceeding":null,"authors":["Yuwei Yan","Yu Shang","Qingbin Zeng","Yu Li","Keyu Zhao","Zhiheng Zheng","Xuefei Ning","Tianji Wu","Shengen Yan","Yu Wang","Fengli Xu","Yong Li"],"abstract":"The AgentSociety Challenge is the first competition in the Web Conference that aims to explore the potential of Large Language Model (LLM) agents in modeling user behavior and enhancing recommender systems on web platforms. The Challenge consists of two tracks: the User Modeling Track and the Recommendation Track. Participants are tasked to utilize a combined dataset from Yelp, Amazon, and Goodreads, along with an interactive environment simulator, to develop innovative LLM agents. The Challenge has attracted 295 teams across the globe and received over 1,400 submissions in total over the course of 37 official competition days. The participants have achieved 21.9% and 20.3% performance improvement for Track 1 and Track 2 in the Development Phase, and 9.1% and 15.9% in the Final Phase, representing a significant accomplishment. This paper discusses the detailed designs of the Challenge, analyzes the outcomes, and highlights the most successful LLM agent designs. To support further research and development, we have open-sourced the benchmark environment at https://tsinghua-fib-lab.github.io/AgentSocietyChallenge.","url_abs":"https://arxiv.org/abs/2502.18754v1","url_pdf":"https://arxiv.org/pdf/2502.18754v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"agentsociety-challenge-designing-llm-agents","repo_url":"https://github.com/tsinghua-fib-lab/agentsocietychallenge","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2502.18754","atlas_url":"https://app.syntology.ai/?focus=2502.18754","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.18754"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tsinghua-fib-lab/agentsocietychallenge","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cdc4ce4d79504560","entry":"calculate_price","repo":"tsinghua-fib-lab/agentsocietychallenge","repo_kind":"official","path":"GTsimulation/ModGTAgent.py","file_url":"https://github.com/tsinghua-fib-lab/agentsocietychallenge/blob/HEAD/GTsimulation/ModGTAgent.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cdc4ce4d79504560"}},{"code_sha256_prefix":"7d558df069ffb851","entry":"check_required_files","repo":"tsinghua-fib-lab/agentsocietychallenge","repo_kind":"official","path":"data_process.py","file_url":"https://github.com/tsinghua-fib-lab/agentsocietychallenge/blob/HEAD/data_process.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d558df069ffb851"}},{"code_sha256_prefix":"ff075553db5b65e2","entry":"filter_data","repo":"tsinghua-fib-lab/agentsocietychallenge","repo_kind":"official","path":"data_process.py","file_url":"https://github.com/tsinghua-fib-lab/agentsocietychallenge/blob/HEAD/data_process.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff075553db5b65e2"}},{"code_sha256_prefix":"c7134037e50f3c8c","entry":"load_data","repo":"tsinghua-fib-lab/agentsocietychallenge","repo_kind":"official","path":"data_process.py","file_url":"https://github.com/tsinghua-fib-lab/agentsocietychallenge/blob/HEAD/data_process.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7134037e50f3c8c"}},{"code_sha256_prefix":"fa222ac81ae15470","entry":"sanitize_input","repo":"tsinghua-fib-lab/agentsocietychallenge","repo_kind":"official","path":"GTsimulation/ModGTAgent.py","file_url":"https://github.com/tsinghua-fib-lab/agentsocietychallenge/blob/HEAD/GTsimulation/ModGTAgent.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa222ac81ae15470"}},{"code_sha256_prefix":"766aece7c07a8cae","entry":"count_tokens","repo":"tsinghua-fib-lab/agentsocietychallenge","repo_kind":"official","path":"GTsimulation/ModGTAgent.py","file_url":"https://github.com/tsinghua-fib-lab/agentsocietychallenge/blob/HEAD/GTsimulation/ModGTAgent.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"766aece7c07a8cae"}},{"code_sha256_prefix":"685037a23906602e","entry":"llm_response","repo":"tsinghua-fib-lab/agentsocietychallenge","repo_kind":"official","path":"GTsimulation/RecGTAgent.py","file_url":"https://github.com/tsinghua-fib-lab/agentsocietychallenge/blob/HEAD/GTsimulation/RecGTAgent.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"685037a23906602e"}},{"code_sha256_prefix":"24ed49c54abe8af9","entry":"num_tokens_from_string","repo":"tsinghua-fib-lab/agentsocietychallenge","repo_kind":"official","path":"GTsimulation/RecGTAgent.py","file_url":"https://github.com/tsinghua-fib-lab/agentsocietychallenge/blob/HEAD/GTsimulation/RecGTAgent.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24ed49c54abe8af9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}