{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/estimating-probability-densities-with","title":"Estimating Probability Densities with Transformer and Denoising Diffusion","arxiv_id":"2407.15703","date":"2024-07-22","proceeding":null,"authors":["Henry W. Leung","Jo Bovy","Joshua S. Speagle"],"abstract":"Transformers are often the go-to architecture to build foundation models that ingest a large amount of training data. But these models do not estimate the probability density distribution when trained on regression problems, yet obtaining full probabilistic outputs is crucial to many fields of science, where the probability distribution of the answer can be non-Gaussian and multimodal. In this work, we demonstrate that training a probabilistic model using a denoising diffusion head on top of the Transformer provides reasonable probability density estimation even for high-dimensional inputs. The combined Transformer+Denoising Diffusion model allows conditioning the output probability density on arbitrary combinations of inputs and it is thus a highly flexible density function emulator of all possible input/output combinations. We illustrate our Transformer+Denoising Diffusion model by training it on a large dataset of astronomical observations and measured labels of stars within our Galaxy and we apply it to a variety of inference tasks to show that the model can infer labels accurately with reasonable distributions.","url_abs":"https://arxiv.org/abs/2407.15703v1","url_pdf":"https://arxiv.org/pdf/2407.15703v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"estimating-probability-densities-with","repo_url":"https://github.com/henrysky/stars_foundation_diffusion","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"density-estimation","task_name":"Density Estimation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.15703","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.15703"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/henrysky/stars_foundation_diffusion","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1e256ca6c383e29b","entry":"get_activation","repo":"henrysky/stars_foundation_diffusion","repo_kind":"official","path":"stellarperceptron/layers.py","file_url":"https://github.com/henrysky/stars_foundation_diffusion/blob/HEAD/stellarperceptron/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e256ca6c383e29b"}},{"code_sha256_prefix":"181136b0f2ef71bb","entry":"get_initializer","repo":"henrysky/stars_foundation_diffusion","repo_kind":"official","path":"stellarperceptron/layers.py","file_url":"https://github.com/henrysky/stars_foundation_diffusion/blob/HEAD/stellarperceptron/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"181136b0f2ef71bb"}},{"code_sha256_prefix":"00ed7823bf5bae82","entry":"robust_mean_squared_error","repo":"henrysky/stars_foundation_diffusion","repo_kind":"official","path":"stellarperceptron/nn_utils.py","file_url":"https://github.com/henrysky/stars_foundation_diffusion/blob/HEAD/stellarperceptron/nn_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00ed7823bf5bae82"}},{"code_sha256_prefix":"cea12a23d8b9ac87","entry":"shuffle_row","repo":"henrysky/stars_foundation_diffusion","repo_kind":"official","path":"stellarperceptron/nn_utils.py","file_url":"https://github.com/henrysky/stars_foundation_diffusion/blob/HEAD/stellarperceptron/nn_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cea12a23d8b9ac87"}},{"code_sha256_prefix":"ac32065953e99b32","entry":"mean_squared_error","repo":"henrysky/stars_foundation_diffusion","repo_kind":"official","path":"stellarperceptron/nn_utils.py","file_url":"https://github.com/henrysky/stars_foundation_diffusion/blob/HEAD/stellarperceptron/nn_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac32065953e99b32"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}