{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/jetformer-an-autoregressive-generative-model","title":"JetFormer: An Autoregressive Generative Model of Raw Images and Text","arxiv_id":"2411.19722","date":"2024-11-29","proceeding":null,"authors":["Michael Tschannen","André Susano Pinto","Alexander Kolesnikov"],"abstract":"Removing modeling constraints and unifying architectures across domains has been a key driver of the recent progress in training large multimodal models. However, most of these models still rely on many separately trained components such as modality-specific encoders and decoders. In this work, we further streamline joint generative modeling of images and text. We propose an autoregressive decoder-only transformer - JetFormer - which is trained to directly maximize the likelihood of raw data, without relying on any separately pretrained components, and can understand and generate both text and images. Specifically, we leverage a normalizing flow model to obtain a soft-token image representation that is jointly trained with an autoregressive multimodal transformer. The normalizing flow model serves as both an image encoder for perception tasks and an image decoder for image generation tasks during inference. JetFormer achieves text-to-image generation quality competitive with recent VQ-VAE- and VAE-based baselines. These baselines rely on pretrained image autoencoders, which are trained with a complex mixture of losses, including perceptual ones. At the same time, JetFormer demonstrates robust image understanding capabilities. To the best of our knowledge, JetFormer is the first model that is capable of generating high-fidelity images and producing strong log-likelihood bounds.","url_abs":"https://arxiv.org/abs/2411.19722v2","url_pdf":"https://arxiv.org/pdf/2411.19722v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"jetformer-an-autoregressive-generative-model","repo_url":"https://github.com/google-research/big_vision","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.19722","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.19722"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/big_vision","reach":null}],"summary":{"ran_draft_wrong":1,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"71393273a1210e33","entry":"text_to_image_retrieval_eval","repo":"google-research/big_vision","repo_kind":"official","path":"big_vision/evaluators/proj/image_text/image_text_retrieval.py","file_url":"https://github.com/google-research/big_vision/blob/HEAD/big_vision/evaluators/proj/image_text/image_text_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"71393273a1210e33"}},{"code_sha256_prefix":"ecdd49907e267cdc","entry":"right_align","repo":"google-research/big_vision","repo_kind":"official","path":"big_vision/models/proj/jetformer/jetformer.py","file_url":"https://github.com/google-research/big_vision/blob/HEAD/big_vision/models/proj/jetformer/jetformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ecdd49907e267cdc"}},{"code_sha256_prefix":"fa422d5f3b0b46d9","entry":"image_to_text_retrieval_eval","repo":"google-research/big_vision","repo_kind":"official","path":"big_vision/evaluators/proj/image_text/image_text_retrieval.py","file_url":"https://github.com/google-research/big_vision/blob/HEAD/big_vision/evaluators/proj/image_text/image_text_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa422d5f3b0b46d9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}