{"url":"/method/hydra","slug":"hydra","name":"Hydra","full_name":"Hydra","full_name_withheld":false,"description_markdown":"**Hydra** is a multi-headed neural network for model distillation with a shared body network. The shared body network learns a joint feature representation that enables each head to capture the predictive behavior of each ensemble member.  Existing distillation methods often train a distillation network to imitate the prediction of a larger network. Hydra instead learns to distill the individual predictions of each ensemble member into separate light-weight head models while amortizing the computation through a shared heavy-weight body network. This retains the diversity of ensemble member predictions which is otherwise lost in knowledge distillation.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2001.04694v2","title":"Hydra: Preserving Ensemble Diversity for Model Distillation","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Knowledge Distillation","url":"/methods/category/knowledge-distillation","pwc_aliases":[]}],"n_papers_tagged":34,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/hydranet-momentum-driven-state-space-duality","title":"HydraNet: Momentum-Driven State Space Duality for Multi-Granularity Tennis Tournaments Analysis","date":"2025-05-28","arxiv_id":"2505.21882","n_code_links":1,"syntology":null},{"paper":null,"title":"Hydra: An Agentic Reasoning Approach for Enhancing Adversarial Robustness and Mitigating Hallucinations in Vision-Language Models","date":"2025-04-19","arxiv_id":"2504.14395","n_code_links":0,"syntology":null},{"paper":null,"title":"A Novel Mamba-based Sequential Recommendation Method","date":"2025-04-10","arxiv_id":"2504.07398","n_code_links":0,"syntology":null},{"paper":null,"title":"Dataset Properties Shape the Success of Neuroimaging-Based Patient Stratification: A Benchmarking Analysis Across Clustering Algorithms","date":"2025-03-15","arxiv_id":"2503.12066","n_code_links":0,"syntology":null},{"paper":"/paper/swift-hydra-self-reinforcing-generative","title":"Swift Hydra: Self-Reinforcing Generative Framework for Anomaly Detection with Multiple Mamba Models","date":"2025-03-09","arxiv_id":"2503.06413","n_code_links":1,"syntology":null},{"paper":null,"title":"An Efficient Attention Mechanism for Sequential Recommendation Tasks: HydraRec","date":"2025-01-02","arxiv_id":"2501.01242","n_code_links":0,"syntology":null},{"paper":null,"title":"Mixture of neural fields for heterogeneous reconstruction in cryo-EM","date":"2024-12-12","arxiv_id":"2412.09420","n_code_links":0,"syntology":null},{"paper":"/paper/sinetra-a-versatile-framework-for-evaluating","title":"SINETRA: a Versatile Framework for Evaluating Single Neuron Tracking in Behaving Animals","date":"2024-11-14","arxiv_id":"2411.09462","n_code_links":1,"syntology":null},{"paper":null,"title":"Rethinking Positive Pairs in Contrastive Learning","date":"2024-10-23","arxiv_id":"2410.18200","n_code_links":0,"syntology":null},{"paper":null,"title":"Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation","date":"2024-09-16","arxiv_id":"2409.10262","n_code_links":0,"syntology":null},{"paper":null,"title":"HYDRA: Hybrid Data Multiplexing and Run-time Layer Configurable DNN Accelerator","date":"2024-09-08","arxiv_id":"2409.04976","n_code_links":0,"syntology":null},{"paper":"/paper/trans-tokenization-and-cross-lingual","title":"Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP","date":"2024-08-08","arxiv_id":"2408.04303","n_code_links":1,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}},{"paper":null,"title":"Flexible Generation of Preference Data for Recommendation Analysis","date":"2024-07-23","arxiv_id":"2407.16594","n_code_links":0,"syntology":null},{"paper":"/paper/hydra-bidirectional-state-space-models","title":"Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers","date":"2024-07-13","arxiv_id":"2407.09941","n_code_links":1,"syntology":{"ran":4,"of":10,"unverified":6,"pointer_only":0}},{"paper":"/paper/m3leo-a-multi-modal-multi-label-earth","title":"M3LEO: A Multi-Modal, Multi-Label Earth Observation Dataset Integrating Interferometric SAR and Multispectral Data","date":"2024-06-06","arxiv_id":"2406.04230","n_code_links":1,"syntology":null},{"paper":"/paper/hydra-model-factorization-framework-for-black","title":"HYDRA: Model Factorization Framework for Black-Box LLM Personalization","date":"2024-06-05","arxiv_id":"2406.02888","n_code_links":1,"syntology":{"ran":16,"of":16,"unverified":0,"pointer_only":16}},{"paper":null,"title":"AnyFit: Controllable Virtual Try-on for Any Combination of Attire Across Any Scenario","date":"2024-05-28","arxiv_id":"2405.18172","n_code_links":0,"syntology":null},{"paper":"/paper/pytorch-ie-fast-and-reproducible-prototyping","title":"PyTorch-IE: Fast and Reproducible Prototyping for Information Extraction","date":"2024-05-16","arxiv_id":"2406.00007","n_code_links":1,"syntology":null},{"paper":"/paper/unleashing-hydra-hybrid-fusion-depth","title":"Unleashing HyDRa: Hybrid Fusion, Depth Consistency and Radar for Unified 3D Perception","date":"2024-03-12","arxiv_id":"2403.07746","n_code_links":1,"syntology":null},{"paper":null,"title":"Hydra: Computer Vision for Data Quality Monitoring","date":"2024-03-01","arxiv_id":"2403.00689","n_code_links":0,"syntology":null},{"paper":"/paper/hydra-sequentially-dependent-draft-heads-for","title":"Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding","date":"2024-02-07","arxiv_id":"2402.05109","n_code_links":1,"syntology":{"ran":8,"of":10,"unverified":2,"pointer_only":0}},{"paper":"/paper/newsreclib-a-pytorch-lightning-library-for","title":"NewsRecLib: A PyTorch-Lightning Library for Neural News Recommendation","date":"2023-10-02","arxiv_id":"2310.01146","n_code_links":1,"syntology":{"ran":4,"of":4,"unverified":0,"pointer_only":0}},{"paper":"/paper/hydra-multi-head-low-rank-adaptation-for","title":"Hydra: Multi-head Low-rank Adaptation for Parameter Efficient Fine-tuning","date":"2023-09-13","arxiv_id":"2309.06922","n_code_links":1,"syntology":null},{"paper":null,"title":"The Hydra Effect: Emergent Self-repair in Language Model Computations","date":"2023-07-28","arxiv_id":"2307.15771","n_code_links":0,"syntology":null},{"paper":null,"title":"Orthogonal-view Microscope for the Biomechanics Investigations of Aquatic Organisms","date":"2023-07-24","arxiv_id":"2307.13079","n_code_links":0,"syntology":null},{"paper":"/paper/regulas-a-bioinformatics-tool-for-the","title":"regulAS: A Bioinformatics Tool for the Integrative Analysis of Alternative Splicing Regulome using RNA-Seq data","date":"2023-07-17","arxiv_id":"2307.08800","n_code_links":1,"syntology":null},{"paper":null,"title":"Non-bilaterians as Model Systems for Tissue Mechanics","date":"2023-06-02","arxiv_id":"2306.01935","n_code_links":0,"syntology":null},{"paper":null,"title":"Efficient Domain Adaptation for Speech Foundation Models","date":"2023-02-03","arxiv_id":"2302.01496","n_code_links":0,"syntology":null},{"paper":"/paper/schnetpack-2-0-a-neural-network-toolbox-for","title":"SchNetPack 2.0: A neural network toolbox for atomistic machine learning","date":"2022-12-11","arxiv_id":"2212.05517","n_code_links":2,"syntology":{"ran":0,"of":4,"unverified":4,"pointer_only":0}},{"paper":null,"title":"Hydra Attention: Efficient Attention with Many Heads","date":"2022-09-15","arxiv_id":"2209.07484","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":4},{"task":"/task/recommendation-systems","name":"Recommendation Systems","papers":4},{"task":"/task/model","name":"model","papers":4},{"task":"/task/benchmarking","name":"Benchmarking","papers":3},{"task":"/task/language-modeling","name":"Language Modeling","papers":3},{"task":"/task/mamba","name":"Mamba","papers":3},{"task":"/task/deep-learning","name":"Deep Learning","papers":2},{"task":"/task/model-selection","name":"Model Selection","papers":2},{"task":"/task/scheduling","name":"Scheduling","papers":2},{"task":"/task/sequential-recommendation","name":"Sequential Recommendation","papers":2},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":2},{"task":"/task/3d-multi-object-tracking","name":"3D Multi-Object Tracking","papers":1},{"task":"/task/3d-object-detection","name":"3D Object Detection","papers":1},{"task":"/task/3d-object-detection-roi","name":"3D Object Detection (RoI)","papers":1},{"task":"/task/3d-reconstruction","name":"3D Reconstruction","papers":1},{"task":"/task/3d-semantic-occupancy-prediction","name":"3D Semantic Occupancy Prediction","papers":1},{"task":"/task/adversarial-attack","name":"Adversarial Attack","papers":1},{"task":"/task/adversarial-defense","name":"Adversarial Defense","papers":1},{"task":"/task/adversarial-robustness","name":"Adversarial Robustness","papers":1},{"task":"/task/anomaly-detection","name":"Anomaly Detection","papers":1}],"tasks_shown":20,"n_tasks":51,"usage_by_year":[{"year":"2020","papers":1},{"year":"2021","papers":2},{"year":"2022","papers":3},{"year":"2023","papers":7},{"year":"2024","papers":15},{"year":"2025","papers":6}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/hydra"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}