{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-bias-variance-covariance-decomposition-of","title":"A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models","arxiv_id":"2310.05833","date":"2023-10-09","proceeding":null,"authors":["Sebastian G. Gruber","Florian Buettner"],"abstract":"Generative models, like large language models, are becoming increasingly relevant in our daily lives, yet a theoretical framework to assess their generalization behavior and uncertainty does not exist. Particularly, the problem of uncertainty estimation is commonly solved in an ad-hoc and task-dependent manner. For example, natural language approaches cannot be transferred to image generation. In this paper, we introduce the first bias-variance-covariance decomposition for kernel scores. This decomposition represents a theoretical framework from which we derive a kernel-based variance and entropy for uncertainty estimation. We propose unbiased and consistent estimators for each quantity which only require generated samples but not the underlying model itself. Based on the wide applicability of kernels, we demonstrate our framework via generalization and uncertainty experiments for image, audio, and language generation. Specifically, kernel entropy for uncertainty estimation is more predictive of performance on CoQA and TriviaQA question answering datasets than existing baselines and can also be applied to closed-source models.","url_abs":"https://arxiv.org/abs/2310.05833v2","url_pdf":"https://arxiv.org/pdf/2310.05833v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-bias-variance-covariance-decomposition-of","repo_url":"https://github.com/mlo-lab/bvcd_generative_models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"triviaqa","task_name":"TriviaQA"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.05833","atlas_url":"https://app.syntology.ai/?focus=2310.05833","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.05833"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mlo-lab/bvcd_generative_models","reach":null}],"summary":{"ran_fixture":2,"ran_honours":1,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"3200c2110ea7265f","entry":"ensemble_block_kernel","repo":"mlo-lab/bvcd_generative_models","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mlo-lab/bvcd_generative_models/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3200c2110ea7265f"}},{"code_sha256_prefix":"7bb9e68f6be9c33f","entry":"ensemble_block_kernel","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"7bb9e68f6be9c33f"}},{"code_sha256_prefix":"0d4cf8fce4830742","entry":"avg_off_diag_blocks","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0d4cf8fce4830742"}},{"code_sha256_prefix":"86d9598f0029780f","entry":"get_diag_blocks","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"86d9598f0029780f"}},{"code_sha256_prefix":"bfcaf41a7ac091d4","entry":"kernel_noise","repo":"mlo-lab/bvcd_generative_models","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mlo-lab/bvcd_generative_models/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfcaf41a7ac091d4"}},{"code_sha256_prefix":"4a47e6ca3ab07146","entry":"avg_off_diag_blocks","repo":"mlo-lab/bvcd_generative_models","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mlo-lab/bvcd_generative_models/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a47e6ca3ab07146"}},{"code_sha256_prefix":"aa2f005203180c22","entry":"bias_estimator","repo":"mlo-lab/bvcd_generative_models","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mlo-lab/bvcd_generative_models/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa2f005203180c22"}},{"code_sha256_prefix":"ce5b3a6c307002e7","entry":"get_diag_blocks","repo":"mlo-lab/bvcd_generative_models","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/mlo-lab/bvcd_generative_models/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce5b3a6c307002e7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}