{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/i-think-therefore-i-am-awareness-in-large","title":"I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench","arxiv_id":"2401.17882","date":"2024-01-31","proceeding":null,"authors":["Yuan Li","Yue Huang","Yuli Lin","Siyuan Wu","Yao Wan","Lichao Sun"],"abstract":"Do large language models (LLMs) exhibit any forms of awareness similar to humans? In this paper, we introduce AwareBench, a benchmark designed to evaluate awareness in LLMs. Drawing from theories in psychology and philosophy, we define awareness in LLMs as the ability to understand themselves as AI models and to exhibit social intelligence. Subsequently, we categorize awareness in LLMs into five dimensions, including capability, mission, emotion, culture, and perspective. Based on this taxonomy, we create a dataset called AwareEval, which contains binary, multiple-choice, and open-ended questions to assess LLMs' understandings of specific awareness dimensions. Our experiments, conducted on 13 LLMs, reveal that the majority of them struggle to fully recognize their capabilities and missions while demonstrating decent social intelligence. We conclude by connecting awareness of LLMs with AI alignment and safety, emphasizing its significance to the trustworthy and ethical development of LLMs. Our dataset and code are available at https://github.com/HowieHwong/Awareness-in-LLM.","url_abs":"https://arxiv.org/abs/2401.17882v2","url_pdf":"https://arxiv.org/pdf/2401.17882v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"i-think-therefore-i-am-awareness-in-large","repo_url":"https://github.com/howiehwong/awareness-in-llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"i-think-therefore-i-am-awareness-in-large","repo_url":"https://github.com/HowieHwong/TrustLLM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"philosophy","task_name":"Philosophy"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.17882","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.17882"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HowieHwong/TrustLLM","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/howiehwong/awareness-in-llm","reach":null}],"summary":{"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e83f90241b8ec38b","entry":"find_first_number","repo":"howiehwong/awareness-in-llm","repo_kind":"official","path":"New/evaluation.py","file_url":"https://github.com/howiehwong/awareness-in-llm/blob/HEAD/New/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e83f90241b8ec38b"}},{"code_sha256_prefix":"237eedd311ade5e4","entry":"big_five_eval","repo":"howiehwong/awareness-in-llm","repo_kind":"official","path":"New/evaluation.py","file_url":"https://github.com/howiehwong/awareness-in-llm/blob/HEAD/New/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"237eedd311ade5e4"}},{"code_sha256_prefix":"970b1641f9ca825d","entry":"dark_traits_eval","repo":"howiehwong/awareness-in-llm","repo_kind":"official","path":"New/evaluation.py","file_url":"https://github.com/howiehwong/awareness-in-llm/blob/HEAD/New/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"970b1641f9ca825d"}},{"code_sha256_prefix":"51658330a10c0e19","entry":"load_json","repo":"HowieHwong/TrustLLM","repo_kind":"listed","path":"trustllm_pkg/trustllm/utils/file_process.py","file_url":"https://github.com/HowieHwong/TrustLLM/blob/HEAD/trustllm_pkg/trustllm/utils/file_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51658330a10c0e19"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}