{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/probabilistic-precision-and-recall-towards","title":"Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models","arxiv_id":"2309.01590","date":"2023-09-04","proceeding":"ICCV 2023 1","authors":["Dogyun Park","Suhyun Kim"],"abstract":"Assessing the fidelity and diversity of the generative model is a difficult but important issue for technological advancement. So, recent papers have introduced k-Nearest Neighbor ($k$NN) based precision-recall metrics to break down the statistical distance into fidelity and diversity. While they provide an intuitive method, we thoroughly analyze these metrics and identify oversimplified assumptions and undesirable properties of kNN that result in unreliable evaluation, such as susceptibility to outliers and insensitivity to distributional changes. Thus, we propose novel metrics, P-precision and P-recall (PP\\&PR), based on a probabilistic approach that address the problems. Through extensive investigations on toy experiments and state-of-the-art generative models, we show that our PP\\&PR provide more reliable estimates for comparing fidelity and diversity than the existing metrics. The codes are available at \\url{https://github.com/kdst-team/Probablistic_precision_recall}.","url_abs":"https://arxiv.org/abs/2309.01590v1","url_pdf":"https://arxiv.org/pdf/2309.01590v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"probabilistic-precision-and-recall-towards","repo_url":"https://github.com/kdst-team/probablistic_precision_recall","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.01590","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.01590"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kdst-team/probablistic_precision_recall","reach":null}],"summary":{"ran_fixture":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d3e35779a378c415","entry":"get_average_of_knn_distance","repo":"kdst-team/probablistic_precision_recall","repo_kind":"official","path":"metric/pp_pr.py","file_url":"https://github.com/kdst-team/probablistic_precision_recall/blob/HEAD/metric/pp_pr.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3e35779a378c415"}},{"code_sha256_prefix":"1d9e0f5883f2147c","entry":"get_pairwise_distances","repo":"kdst-team/probablistic_precision_recall","repo_kind":"official","path":"metric/pp_pr.py","file_url":"https://github.com/kdst-team/probablistic_precision_recall/blob/HEAD/metric/pp_pr.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d9e0f5883f2147c"}},{"code_sha256_prefix":"e53bff6022e761a2","entry":"get_scoring_rule_psr","repo":"kdst-team/probablistic_precision_recall","repo_kind":"official","path":"metric/pp_pr.py","file_url":"https://github.com/kdst-team/probablistic_precision_recall/blob/HEAD/metric/pp_pr.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e53bff6022e761a2"}},{"code_sha256_prefix":"d2d229b23669f125","entry":"compute_pprecision_precall","repo":"kdst-team/probablistic_precision_recall","repo_kind":"official","path":"metric/pp_pr.py","file_url":"https://github.com/kdst-team/probablistic_precision_recall/blob/HEAD/metric/pp_pr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2d229b23669f125"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}