{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-far-are-we-from-believable-ai-agents-a","title":"How Far Are LLMs from Believable AI? A Benchmark for Evaluating the Believability of Human Behavior Simulation","arxiv_id":"2312.17115","date":"2023-12-28","proceeding":null,"authors":["Yang Xiao","Yi Cheng","Jinlan Fu","Jiashuo Wang","Wenjie Li","PengFei Liu"],"abstract":"In recent years, AI has demonstrated remarkable capabilities in simulating human behaviors, particularly those implemented with large language models (LLMs). However, due to the lack of systematic evaluation of LLMs' simulated behaviors, the believability of LLMs among humans remains ambiguous, i.e., it is unclear which behaviors of LLMs are convincingly human-like and which need further improvements. In this work, we design SimulateBench to evaluate the believability of LLMs when simulating human behaviors. In specific, we evaluate the believability of LLMs based on two critical dimensions: 1) consistency: the extent to which LLMs can behave consistently with the given information of a human to simulate; and 2) robustness: the ability of LLMs' simulated behaviors to remain robust when faced with perturbations. SimulateBench includes 65 character profiles and a total of 8,400 questions to examine LLMs' simulated behaviors. Based on SimulateBench, we evaluate the performances of 10 widely used LLMs when simulating characters. The experimental results reveal that current LLMs struggle to align their behaviors with assigned characters and are vulnerable to perturbations in certain factors.","url_abs":"https://arxiv.org/abs/2312.17115v2","url_pdf":"https://arxiv.org/pdf/2312.17115v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-far-are-we-from-believable-ai-agents-a","repo_url":"https://github.com/llmconference/emnlp_conference_2024","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"ai-agent","task_name":"AI Agent"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.17115","atlas_url":"https://app.syntology.ai/?focus=2312.17115","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.17115"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/llmconference/emnlp_conference_2024","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"40481dcd82f7c9a9","entry":"load_json_file","repo":"llmconference/emnlp_conference_2024","repo_kind":"official","path":"benchmark/benchmark_file_util.py","file_url":"https://github.com/llmconference/emnlp_conference_2024/blob/HEAD/benchmark/benchmark_file_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40481dcd82f7c9a9"}},{"code_sha256_prefix":"dc5b4a64081c2586","entry":"process_question","repo":"llmconference/emnlp_conference_2024","repo_kind":"official","path":"benchmark/benchmark_class.py","file_url":"https://github.com/llmconference/emnlp_conference_2024/blob/HEAD/benchmark/benchmark_class.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc5b4a64081c2586"}},{"code_sha256_prefix":"099db8c2586d17b9","entry":"load_prompt","repo":"llmconference/emnlp_conference_2024","repo_kind":"official","path":"benchmark/benchmark_file_util.py","file_url":"https://github.com/llmconference/emnlp_conference_2024/blob/HEAD/benchmark/benchmark_file_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"099db8c2586d17b9"}},{"code_sha256_prefix":"232828d177411eea","entry":"load_template_questions","repo":"llmconference/emnlp_conference_2024","repo_kind":"official","path":"benchmark/benchmark_file_util.py","file_url":"https://github.com/llmconference/emnlp_conference_2024/blob/HEAD/benchmark/benchmark_file_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"232828d177411eea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}