{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-attention-all-nerf-needs","title":"Is Attention All That NeRF Needs?","arxiv_id":"2207.13298","date":"2022-07-27","proceeding":null,"authors":["Mukund Varma T","Peihao Wang","Xuxi Chen","Tianlong Chen","Subhashini Venugopalan","Zhangyang Wang"],"abstract":"We present Generalizable NeRF Transformer (GNT), a transformer-based architecture that reconstructs Neural Radiance Fields (NeRFs) and learns to renders novel views on the fly from source views. While prior works on NeRFs optimize a scene representation by inverting a handcrafted rendering equation, GNT achieves neural representation and rendering that generalizes across scenes using transformers at two stages. (1) The view transformer leverages multi-view geometry as an inductive bias for attention-based scene representation, and predicts coordinate-aligned features by aggregating information from epipolar lines on the neighboring views. (2) The ray transformer renders novel views using attention to decode the features from the view transformer along the sampled points during ray marching. Our experiments demonstrate that when optimized on a single scene, GNT can successfully reconstruct NeRF without an explicit rendering formula due to the learned ray renderer. When trained on multiple scenes, GNT consistently achieves state-of-the-art performance when transferring to unseen scenes and outperform all other methods by ~10% on average. Our analysis of the learned attention maps to infer depth and occlusion indicate that attention enables learning a physically-grounded rendering. Our results show the promise of transformers as a universal modeling tool for graphics. Please refer to our project page for video results: https://vita-group.github.io/GNT/.","url_abs":"https://arxiv.org/abs/2207.13298v3","url_pdf":"https://arxiv.org/pdf/2207.13298v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-attention-all-nerf-needs","repo_url":"https://github.com/vita-group/gnt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"generalizable-novel-view-synthesis","task_name":"Generalizable Novel View Synthesis"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"nerf","task_name":"NeRF"},{"task_slug":"ssim","task_name":"SSIM"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.13298","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.13298"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vita-group/gnt","reach":null}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"listed":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0c8d8666ea836326","entry":"Attention","repo":"vita-group/gnt","repo_kind":"listed","path":"gnt/transformer_network.py","file_url":"https://github.com/vita-group/gnt/blob/HEAD/gnt/transformer_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c8d8666ea836326"}},{"code_sha256_prefix":"274e943335de6884","entry":"Embedder","repo":"vita-group/gnt","repo_kind":"listed","path":"gnt/transformer_network.py","file_url":"https://github.com/vita-group/gnt/blob/HEAD/gnt/transformer_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"274e943335de6884"}},{"code_sha256_prefix":"0793903038531eaf","entry":"FeedForward","repo":"vita-group/gnt","repo_kind":"listed","path":"gnt/transformer_network.py","file_url":"https://github.com/vita-group/gnt/blob/HEAD/gnt/transformer_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0793903038531eaf"}},{"code_sha256_prefix":"2efce5965df73946","entry":"Transformer","repo":"vita-group/gnt","repo_kind":"listed","path":"gnt/transformer_network.py","file_url":"https://github.com/vita-group/gnt/blob/HEAD/gnt/transformer_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2efce5965df73946"}},{"code_sha256_prefix":"92fa9a2a2a001168","entry":"Transformer2D","repo":"vita-group/gnt","repo_kind":"listed","path":"gnt/transformer_network.py","file_url":"https://github.com/vita-group/gnt/blob/HEAD/gnt/transformer_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92fa9a2a2a001168"}},{"code_sha256_prefix":"6d4a16676ba59f68","entry":"Attention2D","repo":"vita-group/gnt","repo_kind":"listed","path":"gnt/transformer_network.py","file_url":"https://github.com/vita-group/gnt/blob/HEAD/gnt/transformer_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d4a16676ba59f68"}},{"code_sha256_prefix":"1ec0574f22121353","entry":"GNT","repo":"vita-group/gnt","repo_kind":"listed","path":"gnt/transformer_network.py","file_url":"https://github.com/vita-group/gnt/blob/HEAD/gnt/transformer_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ec0574f22121353"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}