{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-empirical-study-on-evaluation-metrics-of","title":"An empirical study on evaluation metrics of generative adversarial networks","arxiv_id":"1806.07755","date":"2018-06-19","proceeding":"ICLR 2018 1","authors":["Qiantong Xu","Gao Huang","Yang Yuan","Chuan Guo","Yu Sun","Felix Wu","Kilian Weinberger"],"abstract":"Evaluating generative adversarial networks (GANs) is inherently challenging.\nIn this paper, we revisit several representative sample-based evaluation\nmetrics for GANs, and address the problem of how to evaluate the evaluation\nmetrics. We start with a few necessary conditions for metrics to produce\nmeaningful scores, such as distinguishing real from generated samples,\nidentifying mode dropping and mode collapsing, and detecting overfitting. With\na series of carefully designed experiments, we comprehensively investigate\nexisting sample-based metrics and identify their strengths and limitations in\npractical settings. Based on these results, we observe that kernel Maximum Mean\nDiscrepancy (MMD) and the 1-Nearest-Neighbor (1-NN) two-sample test seem to\nsatisfy most of the desirable properties, provided that the distances between\nsamples are computed in a suitable feature space. Our experiments also unveil\ninteresting properties about the behavior of several popular GAN models, such\nas whether they are memorizing training samples, and how far they are from\nlearning the target distribution.","url_abs":"http://arxiv.org/abs/1806.07755v2","url_pdf":"http://arxiv.org/pdf/1806.07755v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-empirical-study-on-evaluation-metrics-of","repo_url":"https://github.com/xuqiantong/GAN-Metrics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"an-empirical-study-on-evaluation-metrics-of","repo_url":"https://github.com/fabiankaechele/samplingfromautoencoders","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"an-empirical-study-on-evaluation-metrics-of","repo_url":"https://github.com/raahii/evan","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"an-empirical-study-on-evaluation-metrics-of","repo_url":"https://github.com/raahii/video-gans-evaluation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"an-empirical-study-on-evaluation-metrics-of","repo_url":"https://github.com/tagas/vcae","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"convolution","method_name":"Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1806.07755","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1806.07755"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fabiankaechele/samplingfromautoencoders","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/raahii/evan","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tagas/vcae","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xuqiantong/GAN-Metrics","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/raahii/video-gans-evaluation","reach":{"status":"ok"}}],"summary":{"ran_fixture":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"87a2229a391f2f47","entry":"distance","repo":"xuqiantong/GAN-Metrics","repo_kind":"official","path":"framework/metric.py","file_url":"https://github.com/xuqiantong/GAN-Metrics/blob/HEAD/framework/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87a2229a391f2f47"}},{"code_sha256_prefix":"1d68d8df9c84b81f","entry":"knn","repo":"xuqiantong/GAN-Metrics","repo_kind":"official","path":"framework/metric.py","file_url":"https://github.com/xuqiantong/GAN-Metrics/blob/HEAD/framework/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d68d8df9c84b81f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}