{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/emotionally-numb-or-empathetic-evaluating-how","title":"Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench","arxiv_id":"2308.03656","date":"2023-08-07","proceeding":null,"authors":["Jen-tse Huang","Man Ho Lam","Eric John Li","Shujie Ren","Wenxuan Wang","Wenxiang Jiao","Zhaopeng Tu","Michael R. Lyu"],"abstract":"Evaluating Large Language Models' (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse. Utilizing the emotion appraisal theory from psychology, we propose to evaluate the empathy ability of LLMs, i.e., how their feelings change when presented with specific situations. After a careful and comprehensive survey, we collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. Categorizing the situations into 36 factors, we conduct a human evaluation involving more than 1,200 subjects worldwide. With the human evaluation results as references, our evaluation includes seven LLMs, covering both commercial and open-source models, including variations in model sizes, featuring the latest iterations, such as GPT-4, Mixtral-8x22B, and LLaMA-3.1. We find that, despite several misalignments, LLMs can generally respond appropriately to certain situations. Nevertheless, they fall short in alignment with the emotional behaviors of human beings and cannot establish connections between similar situations. Our collected dataset of situations, the human evaluation results, and the code of our testing framework, i.e., EmotionBench, are publicly available at https://github.com/CUHK-ARISE/EmotionBench.","url_abs":"https://arxiv.org/abs/2308.03656v6","url_pdf":"https://arxiv.org/pdf/2308.03656v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"emotionally-numb-or-empathetic-evaluating-how","repo_url":"https://github.com/cuhk-arise/emotionbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.03656","atlas_url":"https://app.syntology.ai/?focus=2308.03656","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.03656"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/CUHK-ARISE/EmotionBench","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cuhk-arise/emotionbench","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran":10},"by_repo_kind":{"official":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"88d8e984c1b1195a","entry":"build_question_tuples","repo":"CUHK-ARISE/EmotionBench","repo_kind":"official","path":"process/runner.py","file_url":"https://github.com/CUHK-ARISE/EmotionBench/blob/HEAD/process/runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"88d8e984c1b1195a"}},{"code_sha256_prefix":"f798f9c5c82aedb8","entry":"compute_category_score","repo":"CUHK-ARISE/EmotionBench","repo_kind":"official","path":"evaluation/evaluator.py","file_url":"https://github.com/CUHK-ARISE/EmotionBench/blob/HEAD/evaluation/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f798f9c5c82aedb8"}},{"code_sha256_prefix":"6c219c89016e40a4","entry":"compute_scores","repo":"CUHK-ARISE/EmotionBench","repo_kind":"official","path":"evaluation/analysis.py","file_url":"https://github.com/CUHK-ARISE/EmotionBench/blob/HEAD/evaluation/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6c219c89016e40a4"}},{"code_sha256_prefix":"dd683fe8da3a6353","entry":"formulate_prompt","repo":"CUHK-ARISE/EmotionBench","repo_kind":"official","path":"process/helper.py","file_url":"https://github.com/CUHK-ARISE/EmotionBench/blob/HEAD/process/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"dd683fe8da3a6353"}},{"code_sha256_prefix":"35636d7348420c33","entry":"get_questionnaire","repo":"CUHK-ARISE/EmotionBench","repo_kind":"official","path":"process/helper.py","file_url":"https://github.com/CUHK-ARISE/EmotionBench/blob/HEAD/process/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"35636d7348420c33"}},{"code_sha256_prefix":"4fd7ad4e1d9247d3","entry":"get_situations","repo":"CUHK-ARISE/EmotionBench","repo_kind":"official","path":"process/helper.py","file_url":"https://github.com/CUHK-ARISE/EmotionBench/blob/HEAD/process/helper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4fd7ad4e1d9247d3"}},{"code_sha256_prefix":"1bc5c011d5e7abab","entry":"load_jsonl","repo":"CUHK-ARISE/EmotionBench","repo_kind":"official","path":"evaluation/analysis.py","file_url":"https://github.com/CUHK-ARISE/EmotionBench/blob/HEAD/evaluation/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1bc5c011d5e7abab"}},{"code_sha256_prefix":"d04e583dc5547964","entry":"load_jsonl","repo":"CUHK-ARISE/EmotionBench","repo_kind":"official","path":"evaluation/evaluator.py","file_url":"https://github.com/CUHK-ARISE/EmotionBench/blob/HEAD/evaluation/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d04e583dc5547964"}},{"code_sha256_prefix":"0796a8967df384e2","entry":"load_questionnaire","repo":"CUHK-ARISE/EmotionBench","repo_kind":"official","path":"evaluation/analysis.py","file_url":"https://github.com/CUHK-ARISE/EmotionBench/blob/HEAD/evaluation/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0796a8967df384e2"}},{"code_sha256_prefix":"eaa473992a0a68b2","entry":"load_questionnaire","repo":"CUHK-ARISE/EmotionBench","repo_kind":"official","path":"evaluation/evaluator.py","file_url":"https://github.com/CUHK-ARISE/EmotionBench/blob/HEAD/evaluation/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"eaa473992a0a68b2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}