{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-evaluation-of-generative-models-in","title":"Evaluating generative models in high energy physics","arxiv_id":"2211.10295","date":"2022-11-18","proceeding":null,"authors":["Raghav Kansal","Anni Li","Javier Duarte","Nadezda Chernyavskaya","Maurizio Pierini","Breno Orzari","Thiago Tomei"],"abstract":"There has been a recent explosion in research into machine-learning-based generative modeling to tackle computational challenges for simulations in high energy physics (HEP). In order to use such alternative simulators in practice, we need well-defined metrics to compare different generative models and evaluate their discrepancy from the true distributions. We present the first systematic review and investigation into evaluation metrics and their sensitivity to failure modes of generative models, using the framework of two-sample goodness-of-fit testing, and their relevance and viability for HEP. Inspired by previous work in both physics and computer vision, we propose two new metrics, the Fr\\'echet and kernel physics distances (FPD and KPD, respectively), and perform a variety of experiments measuring their performance on simple Gaussian-distributed, and simulated high energy jet datasets. We find FPD, in particular, to be the most sensitive metric to all alternative jet distributions tested and recommend its adoption, along with the KPD and Wasserstein distances between individual feature distributions, for evaluating generative models in HEP. We finally demonstrate the efficacy of these proposed metrics in evaluating and comparing a novel attention-based generative adversarial particle transformer to the state-of-the-art message-passing generative adversarial network jet simulation model. The code for our proposed metrics is provided in the open source JetNet Python library.","url_abs":"https://arxiv.org/abs/2211.10295v2","url_pdf":"https://arxiv.org/pdf/2211.10295v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-evaluation-of-generative-models-in","repo_url":"https://github.com/jet-net/jetnet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"on-the-evaluation-of-generative-models-in","repo_url":"https://github.com/rkansal47/MPGAN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"Generative Adversarial Network"},{"task_slug":"high","task_name":"Vocal Bursts Intensity Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.10295","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.10295"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jet-net/jetnet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rkansal47/MPGAN","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":10},"by_repo_kind":{"official":{"samples":11,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bedff51745d2cf84","entry":"l2normalize","repo":"rkansal47/MPGAN","repo_kind":"official","path":"mpgan/spectral_normalization.py","file_url":"https://github.com/rkansal47/MPGAN/blob/HEAD/mpgan/spectral_normalization.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bedff51745d2cf84"}},{"code_sha256_prefix":"967cac1f884cedc4","entry":"augment","repo":"rkansal47/MPGAN","repo_kind":"official","path":"mpgan/augment.py","file_url":"https://github.com/rkansal47/MPGAN/blob/HEAD/mpgan/augment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"967cac1f884cedc4"}},{"code_sha256_prefix":"da3d543e3b015c7a","entry":"get_gen_noise","repo":"rkansal47/MPGAN","repo_kind":"official","path":"train_mnist.py","file_url":"https://github.com/rkansal47/MPGAN/blob/HEAD/train_mnist.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da3d543e3b015c7a"}},{"code_sha256_prefix":"efa4637b9bbdee98","entry":"init_model_dirs","repo":"rkansal47/MPGAN","repo_kind":"official","path":"setup_training.py","file_url":"https://github.com/rkansal47/MPGAN/blob/HEAD/setup_training.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efa4637b9bbdee98"}},{"code_sha256_prefix":"41947e559b23deab","entry":"init_project_dirs","repo":"rkansal47/MPGAN","repo_kind":"official","path":"setup_training.py","file_url":"https://github.com/rkansal47/MPGAN/blob/HEAD/setup_training.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41947e559b23deab"}},{"code_sha256_prefix":"61ec0d308dc1c0a0","entry":"mask_manual","repo":"rkansal47/MPGAN","repo_kind":"official","path":"mpgan/mask_utils.py","file_url":"https://github.com/rkansal47/MPGAN/blob/HEAD/mpgan/mask_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61ec0d308dc1c0a0"}},{"code_sha256_prefix":"835caaadff00a915","entry":"process_args","repo":"rkansal47/MPGAN","repo_kind":"official","path":"setup_training.py","file_url":"https://github.com/rkansal47/MPGAN/blob/HEAD/setup_training.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"835caaadff00a915"}},{"code_sha256_prefix":"afe8c4bee4c1b956","entry":"rand_flip","repo":"rkansal47/MPGAN","repo_kind":"official","path":"mpgan/augment.py","file_url":"https://github.com/rkansal47/MPGAN/blob/HEAD/mpgan/augment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afe8c4bee4c1b956"}},{"code_sha256_prefix":"bf08a7d9e640f1da","entry":"rand_mix","repo":"rkansal47/MPGAN","repo_kind":"official","path":"mpgan/augment.py","file_url":"https://github.com/rkansal47/MPGAN/blob/HEAD/mpgan/augment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf08a7d9e640f1da"}},{"code_sha256_prefix":"6f168d1cadcbd744","entry":"setup_losses","repo":"rkansal47/MPGAN","repo_kind":"official","path":"train_mnist.py","file_url":"https://github.com/rkansal47/MPGAN/blob/HEAD/train_mnist.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f168d1cadcbd744"}},{"code_sha256_prefix":"4919d5ada22da86a","entry":"w1p","repo":"jet-net/jetnet","repo_kind":"official","path":"jetnet/evaluation/gen_metrics.py","file_url":"https://github.com/jet-net/jetnet/blob/HEAD/jetnet/evaluation/gen_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4919d5ada22da86a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}