{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kernel-identification-through-transformers","title":"Kernel Identification Through Transformers","arxiv_id":"2106.08185","date":"2021-06-15","proceeding":"NeurIPS 2021 12","authors":["Fergus Simpson","Ian Davies","Vidhi Lalchand","Alessandro Vullo","Nicolas Durrande","Carl Rasmussen"],"abstract":"Kernel selection plays a central role in determining the performance of Gaussian Process (GP) models, as the chosen kernel determines both the inductive biases and prior support of functions under the GP prior. This work addresses the challenge of constructing custom kernel functions for high-dimensional GP regression models. Drawing inspiration from recent progress in deep learning, we introduce a novel approach named KITT: Kernel Identification Through Transformers. KITT exploits a transformer-based architecture to generate kernel recommendations in under 0.1 seconds, which is several orders of magnitude faster than conventional kernel search algorithms. We train our model using synthetic data generated from priors over a vocabulary of known kernels. By exploiting the nature of the self-attention mechanism, KITT is able to process datasets with inputs of arbitrary dimension. We demonstrate that kernels chosen by KITT yield strong performance over a diverse collection of regression benchmarks.","url_abs":"https://arxiv.org/abs/2106.08185v2","url_pdf":"https://arxiv.org/pdf/2106.08185v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kernel-identification-through-transformers","repo_url":"https://github.com/frgsimpson/kitt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"regression-1","task_name":"regression"}],"methods":[{"method_slug":"gaussian-process","method_name":"Gaussian Process"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.08185","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.08185"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/frgsimpson/kitt","reach":null}],"summary":{"ran":3,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"0ea2ff334d09cbcb","entry":"InducedSetAttentionBlock","repo":"frgsimpson/kitt","repo_kind":"official","path":"kitt/networks/transformer/set_transformer_blocks.py","file_url":"https://github.com/frgsimpson/kitt/blob/HEAD/kitt/networks/transformer/set_transformer_blocks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0ea2ff334d09cbcb"}},{"code_sha256_prefix":"ae35f134435c3169","entry":"MultiHeadAttention","repo":"frgsimpson/kitt","repo_kind":"official","path":"kitt/networks/transformer/set_transformer_blocks.py","file_url":"https://github.com/frgsimpson/kitt/blob/HEAD/kitt/networks/transformer/set_transformer_blocks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ae35f134435c3169"}},{"code_sha256_prefix":"603be4d5243edbb4","entry":"MultiHeadAttentionBlock","repo":"frgsimpson/kitt","repo_kind":"official","path":"kitt/networks/transformer/set_transformer_blocks.py","file_url":"https://github.com/frgsimpson/kitt/blob/HEAD/kitt/networks/transformer/set_transformer_blocks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"603be4d5243edbb4"}},{"code_sha256_prefix":"66755ac81353d809","entry":"scaled_dot_product_attention","repo":"frgsimpson/kitt","repo_kind":"official","path":"kitt/networks/transformer/set_transformer_blocks.py","file_url":"https://github.com/frgsimpson/kitt/blob/HEAD/kitt/networks/transformer/set_transformer_blocks.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"66755ac81353d809"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}