{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-transformers-learn-full-bayesian","title":"Can Transformers Learn Full Bayesian Inference in Context?","arxiv_id":"2501.16825","date":"2025-01-28","proceeding":null,"authors":["Arik Reuter","Tim G. J. Rudner","Vincent Fortuin","David Rügamer"],"abstract":"Transformers have emerged as the dominant architecture in the field of deep learning, with a broad range of applications and remarkable in-context learning (ICL) capabilities. While not yet fully understood, ICL has already proved to be an intriguing phenomenon, allowing transformers to learn in context -- without requiring further training. In this paper, we further advance the understanding of ICL by demonstrating that transformers can perform full Bayesian inference for commonly used statistical models in context. More specifically, we introduce a general framework that builds on ideas from prior fitted networks and continuous normalizing flows which enables us to infer complex posterior distributions for methods such as generalized linear models and latent factor models. Extensive experiments on real-world datasets demonstrate that our ICL approach yields posterior samples that are similar in quality to state-of-the-art MCMC or variational inference methods not operating in context.","url_abs":"https://arxiv.org/abs/2501.16825v1","url_pdf":"https://arxiv.org/pdf/2501.16825v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-transformers-learn-full-bayesian","repo_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"bayesian-inference","task_name":"Bayesian Inference"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"normalizing-flows","method_name":"Normalizing Flows"},{"method_slug":"variational-inference","method_name":"Variational Inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.16825","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.16825"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/arikreuter/icl_for_full_bayesian_inference","reach":null}],"summary":{"ran":11,"ran_violates":1,"unverified":4},"by_repo_kind":{"official":{"samples":16,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"3672477c19ee824f","entry":"ConditionalBatchNorm","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3672477c19ee824f"}},{"code_sha256_prefix":"7334dfff7c15eb8c","entry":"ConditionalLayerNorm","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7334dfff7c15eb8c"}},{"code_sha256_prefix":"8746a274af2a0929","entry":"DecoderBlockConditional","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8746a274af2a0929"}},{"code_sha256_prefix":"cf23a8049b4ef088","entry":"EncoderBlockConditional","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf23a8049b4ef088"}},{"code_sha256_prefix":"8f40ac592744ccad","entry":"Linear_block","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f40ac592744ccad"}},{"code_sha256_prefix":"02ac666b2ac3c6a1","entry":"Linear_skip_block","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"02ac666b2ac3c6a1"}},{"code_sha256_prefix":"67f27ed04e3e9672","entry":"MLP","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67f27ed04e3e9672"}},{"code_sha256_prefix":"19ec286a9acca6f1","entry":"MLPConditional","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"19ec286a9acca6f1"}},{"code_sha256_prefix":"870643f6c036b79d","entry":"PositionalEncoding","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"870643f6c036b79d"}},{"code_sha256_prefix":"c8a3be75f1c48e47","entry":"PositionwiseFeedForward","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8a3be75f1c48e47"}},{"code_sha256_prefix":"a5253f7ea1d3f236","entry":"Rescale","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5253f7ea1d3f236"}},{"code_sha256_prefix":"c838b52183d58395","entry":"string2bool","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"Experiments/RunExperiments/RunExperiments_LFM.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/Experiments/RunExperiments/RunExperiments_LFM.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c838b52183d58395"}},{"code_sha256_prefix":"76c46a50e69c7c97","entry":"TransformerCNF","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76c46a50e69c7c97"}},{"code_sha256_prefix":"420e50cb6931a973","entry":"TransformerConditional","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"420e50cb6931a973"}},{"code_sha256_prefix":"6a80680a5be99f66","entry":"TransformerDecoderConditional","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a80680a5be99f66"}},{"code_sha256_prefix":"ad9007e5c53a09ac","entry":"TransformerEncoderConditional","repo":"arikreuter/icl_for_full_bayesian_inference","repo_kind":"official","path":"LinearRegression/Models/Transformer_CNF.py","file_url":"https://github.com/arikreuter/icl_for_full_bayesian_inference/blob/HEAD/LinearRegression/Models/Transformer_CNF.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad9007e5c53a09ac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}