{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/real-or-robotic-assessing-whether-llms","title":"Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue","arxiv_id":"2409.08330","date":"2024-09-12","proceeding":null,"authors":["Jonathan Ivey","Shivani Kumar","Jiayu Liu","Hua Shen","Sushrita Rakshit","Rohan Raju","Haotian Zhang","Aparna Ananthasubramaniam","Junghwan Kim","Bowen Yi","Dustin Wright","Abraham Israeli","Anders Giovanni Møller","Lechen Zhang","David Jurgens"],"abstract":"Studying and building datasets for dialogue tasks is both expensive and time-consuming due to the need to recruit, train, and collect data from study participants. In response, much recent work has sought to use large language models (LLMs) to simulate both human-human and human-LLM interactions, as they have been shown to generate convincingly human-like text in many settings. However, to what extent do LLM-based simulations \\textit{actually} reflect human dialogues? In this work, we answer this question by generating a large-scale dataset of 100,000 paired LLM-LLM and human-LLM dialogues from the WildChat dataset and quantifying how well the LLM simulations align with their human counterparts. Overall, we find relatively low alignment between simulations and human interactions, demonstrating a systematic divergence along the multiple textual properties, including style and content. Further, in comparisons of English, Chinese, and Russian dialogues, we find that models perform similarly. Our results suggest that LLMs generally perform better when the human themself writes in a way that is more similar to the LLM's own style.","url_abs":"https://arxiv.org/abs/2409.08330v2","url_pdf":"https://arxiv.org/pdf/2409.08330v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"real-or-robotic-assessing-whether-llms","repo_url":"https://github.com/davidjurgens/human-llm-similarity","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.08330","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.08330"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/davidjurgens/human-llm-similarity","reach":{"status":"ok"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"c754f575b037943d","entry":"capital_rate","repo":"davidjurgens/human-llm-similarity","repo_kind":"official","path":"src/analysis/aggregate_metrics_helper.py","file_url":"https://github.com/davidjurgens/human-llm-similarity/blob/HEAD/src/analysis/aggregate_metrics_helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c754f575b037943d"}},{"code_sha256_prefix":"e8a4dc4c8774a07b","entry":"capitalization","repo":"davidjurgens/human-llm-similarity","repo_kind":"official","path":"src/analysis/capitalization_punctuation_similarity.py","file_url":"https://github.com/davidjurgens/human-llm-similarity/blob/HEAD/src/analysis/capitalization_punctuation_similarity.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8a4dc4c8774a07b"}},{"code_sha256_prefix":"86df211b70a3a635","entry":"list_to_dict","repo":"davidjurgens/human-llm-similarity","repo_kind":"official","path":"src/analysis/aggregate_metrics_helper.py","file_url":"https://github.com/davidjurgens/human-llm-similarity/blob/HEAD/src/analysis/aggregate_metrics_helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86df211b70a3a635"}},{"code_sha256_prefix":"c3e35dbeaa8a7e0c","entry":"preprocess_text","repo":"davidjurgens/human-llm-similarity","repo_kind":"official","path":"src/analysis/constituency_parse.py","file_url":"https://github.com/davidjurgens/human-llm-similarity/blob/HEAD/src/analysis/constituency_parse.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3e35dbeaa8a7e0c"}},{"code_sha256_prefix":"e4e2db7551821a66","entry":"punctuation","repo":"davidjurgens/human-llm-similarity","repo_kind":"official","path":"src/analysis/capitalization_punctuation_similarity.py","file_url":"https://github.com/davidjurgens/human-llm-similarity/blob/HEAD/src/analysis/capitalization_punctuation_similarity.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4e2db7551821a66"}},{"code_sha256_prefix":"2a04cf53d184c13b","entry":"run_hf_model","repo":"davidjurgens/human-llm-similarity","repo_kind":"official","path":"src/analysis/metrics_human.py","file_url":"https://github.com/davidjurgens/human-llm-similarity/blob/HEAD/src/analysis/metrics_human.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a04cf53d184c13b"}},{"code_sha256_prefix":"cb587dc94ad775d2","entry":"word_length","repo":"davidjurgens/human-llm-similarity","repo_kind":"official","path":"src/analysis/aggregate_metrics_helper.py","file_url":"https://github.com/davidjurgens/human-llm-similarity/blob/HEAD/src/analysis/aggregate_metrics_helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb587dc94ad775d2"}},{"code_sha256_prefix":"090795ed62d79624","entry":"load_hf_model","repo":"davidjurgens/human-llm-similarity","repo_kind":"official","path":"src/analysis/metrics_human.py","file_url":"https://github.com/davidjurgens/human-llm-similarity/blob/HEAD/src/analysis/metrics_human.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"090795ed62d79624"}},{"code_sha256_prefix":"286ec38deb030950","entry":"sentiment","repo":"davidjurgens/human-llm-similarity","repo_kind":"official","path":"src/analysis/metrics_human.py","file_url":"https://github.com/davidjurgens/human-llm-similarity/blob/HEAD/src/analysis/metrics_human.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"286ec38deb030950"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}