{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/supposedly-equivalent-facts-that-aren-t","title":"Supposedly Equivalent Facts That Aren't? Entity Frequency in Pre-training Induces Asymmetry in LLMs","arxiv_id":"2503.22362","date":"2025-03-28","proceeding":null,"authors":["Yuan He","Bailan He","Zifeng Ding","Alisia Lupidi","Yuqicheng Zhu","Shuo Chen","Caiqi Zhang","Jiaoyan Chen","Yunpu Ma","Volker Tresp","Ian Horrocks"],"abstract":"Understanding and mitigating hallucinations in Large Language Models (LLMs) is crucial for ensuring reliable content generation. While previous research has primarily focused on \"when\" LLMs hallucinate, our work explains \"why\" and directly links model behaviour to the pre-training data that forms their prior knowledge. Specifically, we demonstrate that an asymmetry exists in the recognition of logically equivalent facts, which can be attributed to frequency discrepancies of entities appearing as subjects versus objects. Given that most pre-training datasets are inaccessible, we leverage the fully open-source OLMo series by indexing its Dolma dataset to estimate entity frequencies. Using relational facts (represented as triples) from Wikidata5M, we construct probing datasets to isolate this effect. Our experiments reveal that facts with a high-frequency subject and a low-frequency object are better recognised than their inverse, despite their logical equivalence. The pattern reverses in low-to-high frequency settings, and no statistically significant asymmetry emerges when both entities are high-frequency. These findings highlight the influential role of pre-training data in shaping model predictions and provide insights for inferring the characteristics of pre-training data in closed or partially closed LLMs.","url_abs":"https://arxiv.org/abs/2503.22362v1","url_pdf":"https://arxiv.org/pdf/2503.22362v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"supposedly-equivalent-facts-that-aren-t","repo_url":"https://github.com/KRR-Oxford/FactProbe","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.22362","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.22362"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KRR-Oxford/FactProbe","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"60930369a5fdb810","entry":"analyse_results_for_low_freq_range","repo":"KRR-Oxford/FactProbe","repo_kind":"official","path":"factprobe/utils/analysis.py","file_url":"https://github.com/KRR-Oxford/FactProbe/blob/HEAD/factprobe/utils/analysis.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60930369a5fdb810"}},{"code_sha256_prefix":"fcb77bb252a09fbd","entry":"clean_name","repo":"KRR-Oxford/FactProbe","repo_kind":"official","path":"factprobe/utils/preprocess.py","file_url":"https://github.com/KRR-Oxford/FactProbe/blob/HEAD/factprobe/utils/preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fcb77bb252a09fbd"}},{"code_sha256_prefix":"b11550186c1df048","entry":"clean_names","repo":"KRR-Oxford/FactProbe","repo_kind":"official","path":"factprobe/utils/preprocess.py","file_url":"https://github.com/KRR-Oxford/FactProbe/blob/HEAD/factprobe/utils/preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b11550186c1df048"}},{"code_sha256_prefix":"c17325a144b898f5","entry":"count_correlation","repo":"KRR-Oxford/FactProbe","repo_kind":"official","path":"factprobe/utils/stats.py","file_url":"https://github.com/KRR-Oxford/FactProbe/blob/HEAD/factprobe/utils/stats.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c17325a144b898f5"}},{"code_sha256_prefix":"1ea8c02f05d69fae","entry":"freq_condition","repo":"KRR-Oxford/FactProbe","repo_kind":"official","path":"factprobe/utils/analysis.py","file_url":"https://github.com/KRR-Oxford/FactProbe/blob/HEAD/factprobe/utils/analysis.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ea8c02f05d69fae"}},{"code_sha256_prefix":"936daaad0b92cc75","entry":"freq_dict_from_triple_df","repo":"KRR-Oxford/FactProbe","repo_kind":"official","path":"factprobe/utils/analysis.py","file_url":"https://github.com/KRR-Oxford/FactProbe/blob/HEAD/factprobe/utils/analysis.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"936daaad0b92cc75"}},{"code_sha256_prefix":"4b9b49f7d764a87e","entry":"get_wikidata_count","repo":"KRR-Oxford/FactProbe","repo_kind":"official","path":"factprobe/utils/wikidata.py","file_url":"https://github.com/KRR-Oxford/FactProbe/blob/HEAD/factprobe/utils/wikidata.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b9b49f7d764a87e"}},{"code_sha256_prefix":"56659ea9677a74c6","entry":"is_english_name","repo":"KRR-Oxford/FactProbe","repo_kind":"official","path":"factprobe/utils/preprocess.py","file_url":"https://github.com/KRR-Oxford/FactProbe/blob/HEAD/factprobe/utils/preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56659ea9677a74c6"}},{"code_sha256_prefix":"8e20d69c9043e518","entry":"mcnemar_p","repo":"KRR-Oxford/FactProbe","repo_kind":"official","path":"factprobe/utils/stats.py","file_url":"https://github.com/KRR-Oxford/FactProbe/blob/HEAD/factprobe/utils/stats.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e20d69c9043e518"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}