{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/xt-nested-tokenization-for-larger-context-in","title":"xT: Nested Tokenization for Larger Context in Large Images","arxiv_id":"2403.01915","date":"2024-03-04","proceeding":null,"authors":["Ritwik Gupta","Shufan Li","Tyler Zhu","Jitendra Malik","Trevor Darrell","Karttikeya Mangalam"],"abstract":"Modern computer vision pipelines handle large images in one of two sub-optimal ways: down-sampling or cropping. These two methods incur significant losses in the amount of information and context present in an image. There are many downstream applications in which global context matters as much as high frequency details, such as in real-world satellite imagery; in such cases researchers have to make the uncomfortable choice of which information to discard. We introduce xT, a simple framework for vision transformers which effectively aggregates global context with local details and can model large images end-to-end on contemporary GPUs. We select a set of benchmark datasets across classic vision tasks which accurately reflect a vision model's ability to understand truly large images and incorporate fine details over large scales and assess our method's improvement on them. xT is a streaming, two-stage architecture that adapts existing vision backbones and long sequence language models to effectively model large images without quadratic memory growth. We are able to increase accuracy by up to 8.6% on challenging classification tasks and $F_1$ score by 11.6 on context-dependent segmentation on images as large as 29,000 x 29,000 pixels.","url_abs":"https://arxiv.org/abs/2403.01915v2","url_pdf":"https://arxiv.org/pdf/2403.01915v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"xt-nested-tokenization-for-larger-context-in","repo_url":"https://github.com/bair-climate-initiative/xT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.01915","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.01915"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/bair-climate-initiative/xT","reach":null}],"summary":{"ran":5,"ran_honours":2,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d22a704c40119ed6","entry":"AbstractModel","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d22a704c40119ed6"}},{"code_sha256_prefix":"e3ac9279f9b974ee","entry":"ClassificationDecoder","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e3ac9279f9b974ee"}},{"code_sha256_prefix":"7b7827a7eeaca7c5","entry":"ContextEncoderConfig","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7b7827a7eeaca7c5"}},{"code_sha256_prefix":"5e4a5c0a25c89d9c","entry":"LlamaMLP","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5e4a5c0a25c89d9c"}},{"code_sha256_prefix":"88f7eacf03f1d5b8","entry":"ViTAttention","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"88f7eacf03f1d5b8"}},{"code_sha256_prefix":"57beaaa0fe0fdf0d","entry":"get_2d_sincos_pos_embed","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"57beaaa0fe0fdf0d"}},{"code_sha256_prefix":"8f00b8ca42062354","entry":"get_2d_sincos_pos_embed_base","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8f00b8ca42062354"}},{"code_sha256_prefix":"13afd6063f6e370a","entry":"LLMAttention","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"13afd6063f6e370a"}},{"code_sha256_prefix":"5cc207b2201a86ca","entry":"LLMClassificationDecoder","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5cc207b2201a86ca"}},{"code_sha256_prefix":"a21b33d5b3eaac55","entry":"LLMLayer","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"a21b33d5b3eaac55"}},{"code_sha256_prefix":"a6a3973adcfa45f7","entry":"xT","repo":"bair-climate-initiative/xT","repo_kind":"official","path":"xt/models/decoders/decoder.py","file_url":"https://github.com/bair-climate-initiative/xT/blob/HEAD/xt/models/decoders/decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"a6a3973adcfa45f7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}