{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-03915","title":"Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects","arxiv_id":"2603.03915","date":"2026-03-04","proceeding":null,"authors":["Ji-Lun Peng","Yun-Nung Chen"],"abstract":"Large Language Models (LLMs) have shown remarkable potential in developing role-playing agents (RPAs). However, current evaluation frameworks rely heavily on well-known fictional characters, raising a critical concern: models may be leveraging their internal training memory of these characters rather than demonstrating role-playing capabilities. This reliance often leads to significant performance degradation when RPAs encounter unseen or out-of-distribution personas. To address this, we propose a more rigorous evaluation protocol designed to decouple role-playing proficiency from character recognition. Our experiments across multiple benchmarks demonstrate that anonymizing characters degrades performance, confirming that name exposure provides implicit cues that mask a model's true capability. To mitigate this, we investigate diverse personality augmentation as a method to enhance role fidelity in anonymous settings. We systematically analyze the impact of various personality-description methods on agent behavior and consistency. Our results show that incorporating personality information consistently improves RPA performance. This work establishes a more equitable evaluation standard and validates a scalable, personality-enhanced framework for constructing robust RPAs.","url_abs":"https://arxiv.org/abs/2603.03915","url_pdf":"https://arxiv.org/pdf/2603.03915","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.03915","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.03915"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/MiuLab/RolePlayBench","reach":null}],"summary":{"unverified":8},"by_repo_kind":{"found_in_text":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"816cdfecda5ecb71","entry":"concat_messages","repo":"MiuLab/RolePlayBench","repo_kind":"found_in_text","path":"CharacterEval/get_response.py","file_url":"https://github.com/MiuLab/RolePlayBench/blob/HEAD/CharacterEval/get_response.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"816cdfecda5ecb71"}},{"code_sha256_prefix":"a972bcb38254e1ad","entry":"concat_messages","repo":"MiuLab/RolePlayBench","repo_kind":"found_in_text","path":"CharacterEval/get_response_anonymous.py","file_url":"https://github.com/MiuLab/RolePlayBench/blob/HEAD/CharacterEval/get_response_anonymous.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a972bcb38254e1ad"}},{"code_sha256_prefix":"8f301aa7c29f9221","entry":"format_input","repo":"MiuLab/RolePlayBench","repo_kind":"found_in_text","path":"CharacterEval/run_char_rm.py","file_url":"https://github.com/MiuLab/RolePlayBench/blob/HEAD/CharacterEval/run_char_rm.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f301aa7c29f9221"}},{"code_sha256_prefix":"13a0a9b2c23de2e3","entry":"get_character_info","repo":"MiuLab/RolePlayBench","repo_kind":"found_in_text","path":"InCharacter/code/PDB_character_search.py","file_url":"https://github.com/MiuLab/RolePlayBench/blob/HEAD/InCharacter/code/PDB_character_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"13a0a9b2c23de2e3"}},{"code_sha256_prefix":"e1522eeb965bfec7","entry":"get_personality","repo":"MiuLab/RolePlayBench","repo_kind":"found_in_text","path":"CharacterEval/get_personality.py","file_url":"https://github.com/MiuLab/RolePlayBench/blob/HEAD/CharacterEval/get_personality.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1522eeb965bfec7"}},{"code_sha256_prefix":"bdd7f468b2cb97e7","entry":"judge_16","repo":"MiuLab/RolePlayBench","repo_kind":"found_in_text","path":"InCharacter/code/api_16personality.py","file_url":"https://github.com/MiuLab/RolePlayBench/blob/HEAD/InCharacter/code/api_16personality.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bdd7f468b2cb97e7"}},{"code_sha256_prefix":"1dd3d4585c3b2413","entry":"make_inputs","repo":"MiuLab/RolePlayBench","repo_kind":"found_in_text","path":"CharacterEval/get_response.py","file_url":"https://github.com/MiuLab/RolePlayBench/blob/HEAD/CharacterEval/get_response.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1dd3d4585c3b2413"}},{"code_sha256_prefix":"22fc8cf5816a5765","entry":"submit_16personality_api","repo":"MiuLab/RolePlayBench","repo_kind":"found_in_text","path":"InCharacter/code/api_16personality.py","file_url":"https://github.com/MiuLab/RolePlayBench/blob/HEAD/InCharacter/code/api_16personality.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22fc8cf5816a5765"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}