{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pqmass-probabilistic-assessment-of-the","title":"PQMass: Probabilistic Assessment of the Quality of Generative Models using Probability Mass Estimation","arxiv_id":"2402.04355","date":"2024-02-06","proceeding":null,"authors":["Pablo Lemos","Sammy Sharief","Nikolay Malkin","Salma Salhi","Connor Stone","Laurence Perreault-Levasseur","Yashar Hezaveh"],"abstract":"We propose a likelihood-free method for comparing two distributions given samples from each, with the goal of assessing the quality of generative models. The proposed approach, PQMass, provides a statistically rigorous method for assessing the performance of a single generative model or the comparison of multiple competing models. PQMass divides the sample space into non-overlapping regions and applies chi-squared tests to the number of data samples that fall within each region, giving a p-value that measures the probability that the bin counts derived from two sets of samples are drawn from the same multinomial distribution. PQMass does not depend on assumptions regarding the density of the true distribution, nor does it rely on training or fitting any auxiliary models. We evaluate PQMass on data of various modalities and dimensions, demonstrating its effectiveness in assessing the quality, novelty, and diversity of generated samples. We further show that PQMass scales well to moderately high-dimensional data and thus obviates the need for feature extraction in practical applications.","url_abs":"https://arxiv.org/abs/2402.04355v2","url_pdf":"https://arxiv.org/pdf/2402.04355v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"dimensionality-reduction","task_name":"Dimensionality Reduction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.04355","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.04355"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Ciela-Institute/PQM","reach":null}],"summary":{"ran_draft_wrong":6,"ran_honours":2,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"07da518a36bc7033","entry":"_compute_counts_numpy","repo":"Ciela-Institute/PQM","repo_kind":"found_in_text","path":"src/pqm/pqm.py","file_url":"https://github.com/Ciela-Institute/PQM/blob/HEAD/src/pqm/pqm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07da518a36bc7033"}},{"code_sha256_prefix":"563bfb964db9603b","entry":"_compute_counts_torch","repo":"Ciela-Institute/PQM","repo_kind":"found_in_text","path":"src/pqm/pqm.py","file_url":"https://github.com/Ciela-Institute/PQM/blob/HEAD/src/pqm/pqm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"563bfb964db9603b"}},{"code_sha256_prefix":"d099d92b4a639b9f","entry":"_mean_std_numpy","repo":"Ciela-Institute/PQM","repo_kind":"found_in_text","path":"src/pqm/pqm.py","file_url":"https://github.com/Ciela-Institute/PQM/blob/HEAD/src/pqm/pqm.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d099d92b4a639b9f"}},{"code_sha256_prefix":"322e753b69b8a4c1","entry":"_mean_std_torch","repo":"Ciela-Institute/PQM","repo_kind":"found_in_text","path":"src/pqm/pqm.py","file_url":"https://github.com/Ciela-Institute/PQM/blob/HEAD/src/pqm/pqm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"322e753b69b8a4c1"}},{"code_sha256_prefix":"fc84731c392a9fc2","entry":"_random_permutation","repo":"Ciela-Institute/PQM","repo_kind":"found_in_text","path":"src/pqm/pqm.py","file_url":"https://github.com/Ciela-Institute/PQM/blob/HEAD/src/pqm/pqm.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc84731c392a9fc2"}},{"code_sha256_prefix":"ec0a357e2f3c1754","entry":"_sample_reference_indices_numpy","repo":"Ciela-Institute/PQM","repo_kind":"found_in_text","path":"src/pqm/pqm.py","file_url":"https://github.com/Ciela-Institute/PQM/blob/HEAD/src/pqm/pqm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec0a357e2f3c1754"}},{"code_sha256_prefix":"d40f676ffa5be462","entry":"_sample_reference_indices_torch","repo":"Ciela-Institute/PQM","repo_kind":"found_in_text","path":"src/pqm/pqm.py","file_url":"https://github.com/Ciela-Institute/PQM/blob/HEAD/src/pqm/pqm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d40f676ffa5be462"}},{"code_sha256_prefix":"0cfbef3734c9c0ff","entry":"permute_test","repo":"Ciela-Institute/PQM","repo_kind":"found_in_text","path":"src/pqm/pqm.py","file_url":"https://github.com/Ciela-Institute/PQM/blob/HEAD/src/pqm/pqm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0cfbef3734c9c0ff"}},{"code_sha256_prefix":"058ed9ec51fb1030","entry":"_pqm_test","repo":"Ciela-Institute/PQM","repo_kind":"found_in_text","path":"src/pqm/pqm.py","file_url":"https://github.com/Ciela-Institute/PQM/blob/HEAD/src/pqm/pqm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"058ed9ec51fb1030"}},{"code_sha256_prefix":"b2af5c67809c9b5e","entry":"pqm_pvalue","repo":"Ciela-Institute/PQM","repo_kind":"found_in_text","path":"src/pqm/pqm.py","file_url":"https://github.com/Ciela-Institute/PQM/blob/HEAD/src/pqm/pqm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2af5c67809c9b5e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}