{"url":"/method/dra","slug":"dra","name":"DRA","full_name":"Dynamic Range Activator","full_name_withheld":false,"description_markdown":"Recursive functions with **heteroscedasticity, sparse and high-variance** target distributions introduces a huge complexity that makes their accurate modeling with Neural Networks a difficult task. A main property of recursive maps (e.g factorial function), is their dramatic growth and drop. Learning this recursive behavior requires not only fitting high-frequency patterns within a bounded region but also successfully extrapolating those patterns beyond that region. In time series prediction tasks, capturing periodic even behavior is a challenge. Various methods have been employed to model periodic patterns effectively. However, these approaches typically deal with uni-modal data that also exhibit relatively low variance in both In-Distribution (ID) and Out-Of-Distribution (OOD) regions and do not generalize well to recursive problems with the high-variance observed in our context.\r\nThus, to enable Transformers to capture such behavior and perform proper inference for multi-modal recursive problems, we enhance them by introducing the **Dynamic Range Activator (DRA)**. The DRA is designed to handle the recursive and factorial growth properties inherent in enumerative problems with minimal computational overhead and can be integrated into existing neural networks without requiring significant architectural changes. DRA integrates both harmonic and hyperbolic components as follows,\r\n\\begin{equation}\r\n    \\mathrm{DRA}(x) := x + a \\sin^2\\left(\\frac{x}{b}\\right) + c \\cos(bx) + d \\tanh(bx) \\,,\r\n\\end{equation}\r\nwhere $a, b, c, d$ are learnable parameters. It allows the function to simultaneously model periodic data (through sine and cosine) and rapid growth or attenuation (through the hyperbolic tangent) response.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Can Transformers Do Enumerative Geometry?","paper":"/paper/can-transformers-do-enumerative-geometry","first_author":"Baran Hashemi","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/can-transformers-do-enumerative-geometry"},"source":{"url":"https://arxiv.org/abs/2408.14915v3","title":"Can Transformers Do Enumerative Geometry?","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/Baran-phys/DynamicFormer","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Activation Functions","url":"/methods/category/activation-functions","pwc_aliases":[]}],"n_papers_tagged":4,"archive_num_papers":4,"papers_newest_first":[{"paper":null,"title":"Nosy Layers, Noisy Fixes: Tackling DRAs in Federated Learning Systems using Explainable AI","date":"2025-05-16","arxiv_id":"2505.10942","n_code_links":0,"syntology":null},{"paper":"/paper/dra-grpo-exploring-diversity-aware-reward","title":"DRA-GRPO: Exploring Diversity-Aware Reward Adjustment for R1-Zero-Like Training of Large Language Models","date":"2025-05-14","arxiv_id":"2505.09655","n_code_links":1,"syntology":{"ran":3,"of":8,"unverified":5,"pointer_only":0}},{"paper":null,"title":"Optimizing Antenna Activation for Even Power Distribution in Multi-Beam Satellite Systems Using Genetic Algorithm","date":"2025-02-23","arxiv_id":"2502.16592","n_code_links":0,"syntology":null},{"paper":"/paper/can-transformers-do-enumerative-geometry","title":"Can Transformers Do Enumerative Geometry?","date":"2024-08-27","arxiv_id":"2408.14915","n_code_links":1,"syntology":{"ran":3,"of":3,"unverified":0,"pointer_only":0}}],"papers_shown":4,"tasks":[{"task":"/task/activation-function-synthesis","name":"Activation Function Synthesis","papers":1},{"task":"/task/causal-inference","name":"Causal Inference","papers":1},{"task":"/task/conformal-prediction","name":"Conformal Prediction","papers":1},{"task":"/task/diversity","name":"Diversity","papers":1},{"task":"/task/federated-learning","name":"Federated Learning","papers":1},{"task":"/task/inductive-bias","name":"Inductive Bias","papers":1},{"task":"/task/mathematical-reasoning","name":"Mathematical Reasoning","papers":1},{"task":"/task/time-series-1","name":"Time Series","papers":1},{"task":"/task/regression-1","name":"regression","papers":1},{"task":"/task/scientific-discovery","name":"scientific discovery","papers":1}],"tasks_shown":10,"n_tasks":10,"usage_by_year":[{"year":"2024","papers":1},{"year":"2025","papers":3}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/dra"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}