{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-09989","title":"The System Hallucination Scale (SHS): A Minimal yet Effective Human-Centered Instrument for Evaluating Hallucination-Related Behavior in Large Language Models","arxiv_id":"2603.09989","date":"2026-02-13","proceeding":null,"authors":["Heimo Müller","Dominik Steiger","Markus Plass","Andreas Holzinger"],"abstract":"We introduce the System Hallucination Scale (SHS), a lightweight and human-centered measurement instrument for assessing hallucination-related behavior in large language models (LLMs). Inspired by established psychometric tools such as the System Usability Scale (SUS) and the System Causability Scale (SCS), SHS enables rapid, interpretable, and domain-agnostic evaluation of factual unreliability, incoherence, misleading presentation, and responsiveness to user guidance in model-generated text. SHS is explicitly not an automatic hallucination detector or benchmark metric; instead, it captures how hallucination phenomena manifest from a user perspective under realistic interaction conditions. A real-world evaluation with 210 participants demonstrates high clarity, coherent response behavior, and construct validity, supported by statistical analysis including internal consistency (Cronbach's alpha = 0.87$) and significant inter-dimension correlations (p < 0.001$). Comparative analysis with SUS and SCS reveals complementary measurement properties, supporting SHS as a practical tool for comparative analysis, iterative system development, and deployment monitoring.","url_abs":"https://arxiv.org/abs/2603.09989","url_pdf":"https://arxiv.org/pdf/2603.09989","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2603.09989","atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.09989"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/human-centered-ai-lab/system-hallucination-scale","reach":null}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"9ecbdfa99d5a5102","entry":"DimensionResult","repo":"human-centered-ai-lab/system-hallucination-scale","repo_kind":"found_in_text","path":"python/shs_calculator.py","file_url":"https://github.com/human-centered-ai-lab/system-hallucination-scale/blob/HEAD/python/shs_calculator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9ecbdfa99d5a5102"}},{"code_sha256_prefix":"418f8faa1e6c012a","entry":"SHSResult","repo":"human-centered-ai-lab/system-hallucination-scale","repo_kind":"found_in_text","path":"python/shs_calculator.py","file_url":"https://github.com/human-centered-ai-lab/system-hallucination-scale/blob/HEAD/python/shs_calculator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"418f8faa1e6c012a"}},{"code_sha256_prefix":"2511c300ac45f835","entry":"ConsistencyLevel","repo":"human-centered-ai-lab/system-hallucination-scale","repo_kind":"found_in_text","path":"python/shs_calculator.py","file_url":"https://github.com/human-centered-ai-lab/system-hallucination-scale/blob/HEAD/python/shs_calculator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2511c300ac45f835"}},{"code_sha256_prefix":"55415ef83765ee71","entry":"SHSCalculator","repo":"human-centered-ai-lab/system-hallucination-scale","repo_kind":"found_in_text","path":"python/shs_calculator.py","file_url":"https://github.com/human-centered-ai-lab/system-hallucination-scale/blob/HEAD/python/shs_calculator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"55415ef83765ee71"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}