{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-task-understanding-through","title":"Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses","arxiv_id":"2305.11662","date":"2023-05-19","proceeding":null,"authors":["Xenia Ohmer","Elia Bruni","Dieuwke Hupkes"],"abstract":"At the staggering pace with which the capabilities of large language models (LLMs) are increasing, creating future-proof evaluation sets to assess their understanding becomes more and more challenging. In this paper, we propose a novel paradigm for evaluating LLMs which leverages the idea that correct world understanding should be consistent across different (Fregean) senses of the same meaning. Accordingly, we measure understanding not in terms of correctness but by evaluating consistency across multiple senses that are generated by the model itself. We showcase our approach by instantiating a test where the different senses are different languages, hence using multilingual self-consistency as a litmus test for the model's understanding and simultaneously addressing the important topic of multilinguality. Taking one of the latest versions of ChatGPT as our object of study, we evaluate multilingual consistency for two different tasks across three different languages. We show that its multilingual consistency is still lacking, and that its task and world understanding are thus not language-independent. As our approach does not require any static evaluation corpora in languages other than English, it can easily and cheaply be extended to different languages and tasks and could become an integral part of future benchmarking efforts.","url_abs":"https://arxiv.org/abs/2305.11662v3","url_pdf":"https://arxiv.org/pdf/2305.11662v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-task-understanding-through","repo_url":"https://github.com/xeniaohmer/multisense_consistency","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"form","task_name":"Form"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.11662","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.11662"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xeniaohmer/multisense_consistency","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2d719f66d1260a21","entry":"cut_double_quotes","repo":"xeniaohmer/multisense_consistency","repo_kind":"official","path":"utils/response_standardization.py","file_url":"https://github.com/xeniaohmer/multisense_consistency/blob/HEAD/utils/response_standardization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d719f66d1260a21"}},{"code_sha256_prefix":"3152f799f82f4e0c","entry":"get_table","repo":"xeniaohmer/multisense_consistency","repo_kind":"official","path":"utils/analysis_helpers.py","file_url":"https://github.com/xeniaohmer/multisense_consistency/blob/HEAD/utils/analysis_helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3152f799f82f4e0c"}},{"code_sha256_prefix":"bb2168f324aab034","entry":"load_standardized_response","repo":"xeniaohmer/multisense_consistency","repo_kind":"official","path":"utils/analysis_helpers.py","file_url":"https://github.com/xeniaohmer/multisense_consistency/blob/HEAD/utils/analysis_helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb2168f324aab034"}},{"code_sha256_prefix":"71476d0db50aeff1","entry":"standardize_classification_response","repo":"xeniaohmer/multisense_consistency","repo_kind":"official","path":"utils/response_standardization.py","file_url":"https://github.com/xeniaohmer/multisense_consistency/blob/HEAD/utils/response_standardization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71476d0db50aeff1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}