{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/assessing-generative-models-via-precision-and","title":"Assessing Generative Models via Precision and Recall","arxiv_id":"1806.00035","date":"2018-05-31","proceeding":"NeurIPS 2018 12","authors":["Mehdi S. M. Sajjadi","Olivier Bachem","Mario Lucic","Olivier Bousquet","Sylvain Gelly"],"abstract":"Recent advances in generative modeling have led to an increased interest in\nthe study of statistical divergences as means of model comparison. Commonly\nused evaluation methods, such as the Frechet Inception Distance (FID),\ncorrelate well with the perceived quality of samples and are sensitive to mode\ndropping. However, these metrics are unable to distinguish between different\nfailure cases since they only yield one-dimensional scores. We propose a novel\ndefinition of precision and recall for distributions which disentangles the\ndivergence into two separate dimensions. The proposed notion is intuitive,\nretains desirable properties, and naturally leads to an efficient algorithm\nthat can be used to evaluate generative models. We relate this notion to total\nvariation as well as to recent evaluation metrics such as Inception Score and\nFID. To demonstrate the practical utility of the proposed approach we perform\nan empirical study on several variants of Generative Adversarial Networks and\nVariational Autoencoders. In an extensive set of experiments we show that the\nproposed metric is able to disentangle the quality of generated samples from\nthe coverage of the target distribution.","url_abs":"http://arxiv.org/abs/1806.00035v2","url_pdf":"http://arxiv.org/pdf/1806.00035v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"assessing-generative-models-via-precision-and","repo_url":"https://github.com/msmsajjadi/precision-recall-distributions","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"assessing-generative-models-via-precision-and","repo_url":"https://github.com/ok1zjf/lbae","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"assessing-generative-models-via-precision-and","repo_url":"https://github.com/raahii/evan","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"assessing-generative-models-via-precision-and","repo_url":"https://github.com/raahii/video-gans-evaluation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1806.00035","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1806.00035"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ok1zjf/lbae","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/raahii/evan","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/msmsajjadi/precision-recall-distributions","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/raahii/video-gans-evaluation","reach":{"status":"ok"}}],"summary":{"ran_fixture":4,"unverified":11},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":12,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5277c96076ea2fe5","entry":"calculate_frechet_distance","repo":"ok1zjf/lbae","repo_kind":"listed","path":"fid.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/fid.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5277c96076ea2fe5"}},{"code_sha256_prefix":"b9da61c36d3b8f11","entry":"compute_prd","repo":"msmsajjadi/precision-recall-distributions","repo_kind":"official","path":"prd_score.py","file_url":"https://github.com/msmsajjadi/precision-recall-distributions/blob/HEAD/prd_score.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b9da61c36d3b8f11"}},{"code_sha256_prefix":"5d84440e02c3a502","entry":"compute_prd_from_embedding","repo":"msmsajjadi/precision-recall-distributions","repo_kind":"official","path":"prd_score.py","file_url":"https://github.com/msmsajjadi/precision-recall-distributions/blob/HEAD/prd_score.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d84440e02c3a502"}},{"code_sha256_prefix":"5adbefd0ebfa3e5e","entry":"prd_to_max_f_beta_pair","repo":"msmsajjadi/precision-recall-distributions","repo_kind":"official","path":"prd_score.py","file_url":"https://github.com/msmsajjadi/precision-recall-distributions/blob/HEAD/prd_score.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5adbefd0ebfa3e5e"}},{"code_sha256_prefix":"664d8c6c98c4ce7c","entry":"calculate_activation_statistics","repo":"ok1zjf/lbae","repo_kind":"listed","path":"fid.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/fid.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"664d8c6c98c4ce7c"}},{"code_sha256_prefix":"0bee687e2a3cbee8","entry":"corrupt","repo":"ok1zjf/lbae","repo_kind":"listed","path":"datasets.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bee687e2a3cbee8"}},{"code_sha256_prefix":"579daa09f4887be8","entry":"get_activations","repo":"ok1zjf/lbae","repo_kind":"listed","path":"fid.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/fid.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"579daa09f4887be8"}},{"code_sha256_prefix":"5a075ff84473359d","entry":"nearestPDX","repo":"ok1zjf/lbae","repo_kind":"listed","path":"sampler.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/sampler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a075ff84473359d"}},{"code_sha256_prefix":"13d8e00ecc2023db","entry":"norm","repo":"ok1zjf/lbae","repo_kind":"listed","path":"sampler.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/sampler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13d8e00ecc2023db"}},{"code_sha256_prefix":"8cdf564a25c20149","entry":"psnr","repo":"ok1zjf/lbae","repo_kind":"listed","path":"image_utils.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/image_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cdf564a25c20149"}},{"code_sha256_prefix":"1bd4c31f00d84b70","entry":"run_command","repo":"ok1zjf/lbae","repo_kind":"listed","path":"sys_utils.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/sys_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bd4c31f00d84b70"}},{"code_sha256_prefix":"afc8d137b43810a4","entry":"save_image","repo":"ok1zjf/lbae","repo_kind":"listed","path":"image_utils.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/image_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afc8d137b43810a4"}},{"code_sha256_prefix":"239d3d651c2bea69","entry":"ssim","repo":"ok1zjf/lbae","repo_kind":"listed","path":"image_utils.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/image_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"239d3d651c2bea69"}},{"code_sha256_prefix":"0047ef4dfbce17cc","entry":"todeg","repo":"ok1zjf/lbae","repo_kind":"listed","path":"sampler.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/sampler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0047ef4dfbce17cc"}},{"code_sha256_prefix":"c232fb314605487f","entry":"tohms","repo":"ok1zjf/lbae","repo_kind":"listed","path":"sys_utils.py","file_url":"https://github.com/ok1zjf/lbae/blob/HEAD/sys_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c232fb314605487f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}