{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/moca-measuring-human-language-model-alignment-1","title":"MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks","arxiv_id":"2310.19677","date":"2023-10-30","proceeding":"NeurIPS 2023 11","authors":["Allen Nie","Yuhui Zhang","Atharva Amdekar","Chris Piech","Tatsunori Hashimoto","Tobias Gerstenberg"],"abstract":"Human commonsense understanding of the physical and social world is organized around intuitive theories. These theories support making causal and moral judgments. When something bad happens, we naturally ask: who did what, and why? A rich literature in cognitive science has studied people's causal and moral intuitions. This work has revealed a number of factors that systematically influence people's judgments, such as the violation of norms and whether the harm is avoidable or inevitable. We collected a dataset of stories from 24 cognitive science papers and developed a system to annotate each story with the factors they investigated. Using this dataset, we test whether large language models (LLMs) make causal and moral judgments about text-based scenarios that align with those of human participants. On the aggregate level, alignment has improved with more recent LLMs. However, using statistical analyses, we find that LLMs weigh the different factors quite differently from human participants. These results show how curated, challenge datasets combined with insights from cognitive science can help us go beyond comparisons based merely on aggregate metrics: we uncover LLMs implicit tendencies and show to what extent these align with human intuitions.","url_abs":"https://arxiv.org/abs/2310.19677v2","url_pdf":"https://arxiv.org/pdf/2310.19677v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.19677","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.19677"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/cicl-stanford/moca","reach":{"status":"ok"}}],"summary":{"ran":11},"by_repo_kind":{"found_in_text":{"samples":11,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"43089ea4946d080c","entry":"agent_awareness_translator","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"moca/thought_as_text_translator.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/moca/thought_as_text_translator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43089ea4946d080c"}},{"code_sha256_prefix":"87c4261e86afefb8","entry":"bca_bootstrap","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"experiments/causal_effect_analysis.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/experiments/causal_effect_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87c4261e86afefb8"}},{"code_sha256_prefix":"fd056bb65f483c64","entry":"bootstrap","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"experiments/causal_effect_analysis.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/experiments/causal_effect_analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd056bb65f483c64"}},{"code_sha256_prefix":"8c488dd3c0773feb","entry":"causal_structure_translator","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"moca/thought_as_text_translator.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/moca/thought_as_text_translator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c488dd3c0773feb"}},{"code_sha256_prefix":"3c62ef7ada1437a6","entry":"closest_value","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"moca/evaluator.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/moca/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c62ef7ada1437a6"}},{"code_sha256_prefix":"a730544aeb5a5ad0","entry":"compute_acme_radar","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"experiments/amce_radar_stats.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/experiments/amce_radar_stats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a730544aeb5a5ad0"}},{"code_sha256_prefix":"69c1143dd064286c","entry":"cross_entropy","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"moca/evaluator.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/moca/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69c1143dd064286c"}},{"code_sha256_prefix":"595ec11c1939ae03","entry":"event_normality_translator","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"moca/thought_as_text_translator.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/moca/thought_as_text_translator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"595ec11c1939ae03"}},{"code_sha256_prefix":"824f5cf99170c31e","entry":"redirect_to_base_dir","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"moca/prompt.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/moca/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"824f5cf99170c31e"}},{"code_sha256_prefix":"6258ee62d8160d98","entry":"remove_victm_quant","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"moca/prompt.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/moca/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6258ee62d8160d98"}},{"code_sha256_prefix":"55438b793c3f6279","entry":"rephrase_victim","repo":"cicl-stanford/moca","repo_kind":"found_in_text","path":"moca/prompt.py","file_url":"https://github.com/cicl-stanford/moca/blob/HEAD/moca/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55438b793c3f6279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}