{"url":"/method/graddrop","slug":"graddrop","name":"GradDrop","full_name":"Gradient Sign Dropout","full_name_withheld":false,"description_markdown":"**GradDrop**, or **Gradient Sign Dropout**, is a probabilistic masking procedure which samples gradients at an activation layer based on their level of consistency. It is applied as a layer in any standard network forward pass, usually on the final layer before the prediction head to save on compute overhead and maximize benefits during backpropagation. Below, we develop the GradDrop formalism. Throughout, o denotes elementwise multiplication after any necessary tiling operations (if any) are completed.\r\nTo implement GradDrop, we first define the Gradient Positive Sign Purity, $\\mathcal{P}$, as\r\n\r\n$$\r\n\\mathcal{P}=\\frac{1}{2}\\left(1+\\frac{\\sum\\_{i} \\nabla L_\\{i}}{\\sum\\_{i}\\left|\\nabla L\\_{i}\\right|}\\right)\r\n$$\r\n\r\n$\\mathcal{P}$ is bounded by $[0,1] .$ For multiple gradient values $\\nabla\\_{a} L\\_{i}$ at some scalar $a$, we see that $\\mathcal{P}=0$ if $\\nabla_{a} L\\_{i}<0 $ $\\forall i$, while $\\mathcal{P}=1$ if $\\nabla\\_{a} L\\_{i}>0$ $\\forall i $. Thus, $\\mathcal{P}$ is a measure of how many positive gradients are present at any given value. We then form a mask for each gradient $\\mathcal{M}\\_{i}$ as follows:\r\n\r\n$$\r\n\\mathcal{M}\\_{i}=\\mathcal{I}[f(\\mathcal{P})>U] \\circ \\mathcal{I}\\left[\\nabla L\\_{i}>0\\right]+\\mathcal{I}[f(\\mathcal{P})