{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beware-of-the-simulated-dag-varsortability-in","title":"Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To Game","arxiv_id":"2102.13647","date":"2021-02-26","proceeding":"NeurIPS 2021 12","authors":["Alexander G. Reisach","Christof Seiler","Sebastian Weichwald"],"abstract":"Simulated DAG models may exhibit properties that, perhaps inadvertently, render their structure identifiable and unexpectedly affect structure learning algorithms. Here, we show that marginal variance tends to increase along the causal order for generically sampled additive noise models. We introduce varsortability as a measure of the agreement between the order of increasing marginal variance and the causal order. For commonly sampled graphs and model parameters, we show that the remarkable performance of some continuous structure learning algorithms can be explained by high varsortability and matched by a simple baseline method. Yet, this performance may not transfer to real-world data where varsortability may be moderate or dependent on the choice of measurement scales. On standardized data, the same algorithms fail to identify the ground-truth DAG or its Markov equivalence class. While standardization removes the pattern in marginal variance, we show that data generating processes that incur high varsortability also leave a distinct covariance pattern that may be exploited even after standardization. Our findings challenge the significance of generic benchmarks with independently drawn parameters. The code is available at https://github.com/Scriddie/Varsortability.","url_abs":"https://arxiv.org/abs/2102.13647v3","url_pdf":"https://arxiv.org/pdf/2102.13647v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beware-of-the-simulated-dag-varsortability-in","repo_url":"https://github.com/scriddie/varsortability","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"beware-of-the-simulated-dag-varsortability-in","repo_url":"https://github.com/vzantedeschi/daguerreotype","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"causal-discovery","task_name":"Causal Discovery"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2102.13647","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2102.13647"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Scriddie/Varsortability","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/scriddie/varsortability","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vzantedeschi/daguerreotype","reach":null}],"summary":{"ran_draft_wrong":2,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c2ba12495ea1e34a","entry":"coefficient_based_order","repo":"scriddie/varsortability","repo_kind":"official","path":"src/chain_orientation.py","file_url":"https://github.com/scriddie/varsortability/blob/HEAD/src/chain_orientation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2ba12495ea1e34a"}},{"code_sha256_prefix":"019be0bd17448f55","entry":"get_variances","repo":"vzantedeschi/daguerreotype","repo_kind":"listed","path":"daguerreo/utils.py","file_url":"https://github.com/vzantedeschi/daguerreotype/blob/HEAD/daguerreo/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"019be0bd17448f55"}},{"code_sha256_prefix":"7976785b32d28df1","entry":"variance_based_order","repo":"scriddie/varsortability","repo_kind":"official","path":"src/chain_orientation.py","file_url":"https://github.com/scriddie/varsortability/blob/HEAD/src/chain_orientation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7976785b32d28df1"}},{"code_sha256_prefix":"ac90314a0b2a1492","entry":"sortnregress","repo":"scriddie/varsortability","repo_kind":"official","path":"src/sortnregress.py","file_url":"https://github.com/scriddie/varsortability/blob/HEAD/src/sortnregress.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac90314a0b2a1492"}},{"code_sha256_prefix":"82949ad32ef09d09","entry":"varsortability","repo":"scriddie/varsortability","repo_kind":"official","path":"src/varsortability.py","file_url":"https://github.com/scriddie/varsortability/blob/HEAD/src/varsortability.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82949ad32ef09d09"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}