{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hydravit-stacking-heads-for-a-scalable-vit","title":"HydraViT: Stacking Heads for a Scalable ViT","arxiv_id":"2409.17978","date":"2024-09-26","proceeding":null,"authors":["Janek Haberer","Ali Hojjat","Olaf Landsiedel"],"abstract":"The architecture of Vision Transformers (ViTs), particularly the Multi-head Attention (MHA) mechanism, imposes substantial hardware demands. Deploying ViTs on devices with varying constraints, such as mobile phones, requires multiple models of different sizes. However, this approach has limitations, such as training and storing each required model separately. This paper introduces HydraViT, a novel approach that addresses these limitations by stacking attention heads to achieve a scalable ViT. By repeatedly changing the size of the embedded dimensions throughout each layer and their corresponding number of attention heads in MHA during training, HydraViT induces multiple subnetworks. Thereby, HydraViT achieves adaptability across a wide spectrum of hardware environments while maintaining performance. Our experimental results demonstrate the efficacy of HydraViT in achieving a scalable ViT with up to 10 subnetworks, covering a wide range of resource constraints. HydraViT achieves up to 5 p.p. more accuracy with the same GMACs and up to 7 p.p. more accuracy with the same throughput on ImageNet-1K compared to the baselines, making it an effective solution for scenarios where hardware availability is diverse or varies over time. Source code available at https://github.com/ds-kiel/HydraViT.","url_abs":"https://arxiv.org/abs/2409.17978v2","url_pdf":"https://arxiv.org/pdf/2409.17978v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hydravit-stacking-heads-for-a-scalable-vit","repo_url":"https://github.com/ds-kiel/hydravit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.17978","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.17978"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ds-kiel/HydraViT","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ds-kiel/hydravit","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":14,"unverified":2},"by_repo_kind":{"official":{"samples":16,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"39934307038ef5a9","entry":"adaptive_avgmax_pool2d","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/adaptive_avgmax_pool.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/adaptive_avgmax_pool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39934307038ef5a9"}},{"code_sha256_prefix":"fa9e3a5ceeac7703","entry":"adaptive_catavgmax_pool2d","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/adaptive_avgmax_pool.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/adaptive_avgmax_pool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa9e3a5ceeac7703"}},{"code_sha256_prefix":"13295c0de28cc1f1","entry":"adaptive_pool_feat_mult","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/adaptive_avgmax_pool.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/adaptive_avgmax_pool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13295c0de28cc1f1"}},{"code_sha256_prefix":"a0b1f77ef449514b","entry":"checkpoint_filter_fn","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/models/hydravit.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/models/hydravit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0b1f77ef449514b"}},{"code_sha256_prefix":"0bbab6441363adb9","entry":"get_init_weights_vit","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/models/hydravit.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/models/hydravit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bbab6441363adb9"}},{"code_sha256_prefix":"91c27523dde7eb4a","entry":"hard_sigmoid_jit","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/activations_jit.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/activations_jit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91c27523dde7eb4a"}},{"code_sha256_prefix":"801c85b8903bfff5","entry":"mish","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/activations.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"801c85b8903bfff5"}},{"code_sha256_prefix":"14eddd8cc4c65fe0","entry":"mish_jit","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/activations_jit.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/activations_jit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"14eddd8cc4c65fe0"}},{"code_sha256_prefix":"d83b6ecf3cdc1d6a","entry":"sigmoid","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/activations.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d83b6ecf3cdc1d6a"}},{"code_sha256_prefix":"717e4d22b6d103e6","entry":"swish","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/activations.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"717e4d22b6d103e6"}},{"code_sha256_prefix":"664ed3a2c3b8ab90","entry":"swish_bwd","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/activations_me.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/activations_me.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"664ed3a2c3b8ab90"}},{"code_sha256_prefix":"55d18267472fb7c9","entry":"swish_fwd","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/activations_me.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/activations_me.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55d18267472fb7c9"}},{"code_sha256_prefix":"8e341f8585b3fe94","entry":"swish_jit","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/activations_jit.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/activations_jit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e341f8585b3fe94"}},{"code_sha256_prefix":"7d1389c0f75235c7","entry":"swish_me","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/layers/activations_me.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/layers/activations_me.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d1389c0f75235c7"}},{"code_sha256_prefix":"ed04551ca8295bde","entry":"resize_pos_embed","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"timm/models/hydravit.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/timm/models/hydravit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed04551ca8295bde"}},{"code_sha256_prefix":"e826d2b8a200b367","entry":"validate","repo":"ds-kiel/HydraViT","repo_kind":"official","path":"validate.py","file_url":"https://github.com/ds-kiel/HydraViT/blob/HEAD/validate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e826d2b8a200b367"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}