{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-15892","title":"Temporal Fact Conflicts in LLMs: Reproducibility Insights from Unifying DYNAMICQA and MULAN","arxiv_id":"2603.15892","date":"2026-03-16","proceeding":null,"authors":["Ritajit Dey","Iadh Ounis","Graham McDonald","Yashar Moshfeghi"],"abstract":"Large Language Models (LLMs) often struggle with temporal fact conflicts due to outdated or evolving information in their training data. Two recent studies with accompanying datasets report opposite conclusions on whether external context can effectively resolve such conflicts. DYNAMICQA evaluates how effective external context is in shifting the model's output distribution, finding that temporal facts are more resistant to change. In contrast, MULAN examines how often external context changes memorised facts, concluding that temporal facts are easier to update. In this reproducibility paper, we first reproduce experiments from both benchmarks. We then reproduce the experiments of each study on the dataset of the other to investigate the source of their disagreement. To enable direct comparison of findings, we standardise both datasets to align with the evaluation settings of each study. Importantly, using an LLM, we synthetically generate realistic natural language contexts to replace MULAN's programmatically constructed statements when reproducing the findings of DYNAMICQA. Our analysis reveals strong dataset dependence: MULAN's findings generalise under both methodological frameworks, whereas applying MULAN's evaluation to DYNAMICQA yields mixed outcomes. Finally, while the original studies only considered 7B LLMs, we reproduce these experiments across LLMs of varying sizes, revealing how model size influences the encoding and updating of temporal facts. Our results highlight how dataset design, evaluation metrics, and model size shape LLM behaviour in the presence of temporal knowledge conflicts.","url_abs":"https://arxiv.org/abs/2603.15892","url_pdf":"https://arxiv.org/pdf/2603.15892","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.15892","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.15892"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/terrierteam/temporal_conflicts","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"found_in_text":{"samples":9,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"terrierteam/temporal_conflicts","repo_kind":"found_in_text","path":"fact_mutability/analysis/f1_score.py","file_url":"https://github.com/terrierteam/temporal_conflicts/blob/HEAD/fact_mutability/analysis/f1_score.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"fd5f06fcb775f9e0","entry":"exact_match_score","repo":"terrierteam/temporal_conflicts","repo_kind":"found_in_text","path":"fact_mutability/analysis/f1_score.py","file_url":"https://github.com/terrierteam/temporal_conflicts/blob/HEAD/fact_mutability/analysis/f1_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd5f06fcb775f9e0"}},{"code_sha256_prefix":"96e55856ac1587bd","entry":"f1_score","repo":"terrierteam/temporal_conflicts","repo_kind":"found_in_text","path":"fact_mutability/analysis/f1_score.py","file_url":"https://github.com/terrierteam/temporal_conflicts/blob/HEAD/fact_mutability/analysis/f1_score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96e55856ac1587bd"}},{"code_sha256_prefix":"990a2f6afb605b7b","entry":"get_generation_config","repo":"terrierteam/temporal_conflicts","repo_kind":"found_in_text","path":"fact_mutability/bkp_inference.py","file_url":"https://github.com/terrierteam/temporal_conflicts/blob/HEAD/fact_mutability/bkp_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"990a2f6afb605b7b"}},{"code_sha256_prefix":"2a6131b16a998443","entry":"get_scores","repo":"terrierteam/temporal_conflicts","repo_kind":"found_in_text","path":"fact_mutability/bkp_inference.py","file_url":"https://github.com/terrierteam/temporal_conflicts/blob/HEAD/fact_mutability/bkp_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a6131b16a998443"}},{"code_sha256_prefix":"dd0ebdab32113a50","entry":"load_aliases","repo":"terrierteam/temporal_conflicts","repo_kind":"found_in_text","path":"fact_mutability/evaluation.py","file_url":"https://github.com/terrierteam/temporal_conflicts/blob/HEAD/fact_mutability/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd0ebdab32113a50"}},{"code_sha256_prefix":"8badf1e08d4e9f6d","entry":"load_queries","repo":"terrierteam/temporal_conflicts","repo_kind":"found_in_text","path":"fact_mutability/evaluation.py","file_url":"https://github.com/terrierteam/temporal_conflicts/blob/HEAD/fact_mutability/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8badf1e08d4e9f6d"}},{"code_sha256_prefix":"087a8df73a06c291","entry":"prepare_prompt","repo":"terrierteam/temporal_conflicts","repo_kind":"found_in_text","path":"fact_mutability/bkp_inference.py","file_url":"https://github.com/terrierteam/temporal_conflicts/blob/HEAD/fact_mutability/bkp_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"087a8df73a06c291"}},{"code_sha256_prefix":"0de3ad231f1c81d1","entry":"process_vllm_output","repo":"terrierteam/temporal_conflicts","repo_kind":"found_in_text","path":"fact_mutability/inference_vllm.py","file_url":"https://github.com/terrierteam/temporal_conflicts/blob/HEAD/fact_mutability/inference_vllm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0de3ad231f1c81d1"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}