G-GLN Neuron

A G-GLN Neuron is a type of neuron used in the G-GLN architecture. G-GLN. The key idea is that further representational power can be added to a weighted product of Gaussians via a contextual gating procedure. This is achieved by extending a weighted product of Gaussians model with an additional type of input called side information. The side information will be used by a neuron to select a weight vector to apply for a given example from a table of weight vectors. In typical applications to regression, the side information is defined as the (normalized) input features for an input example: i.e. z=(x-x̅) / σₓ.

More formally, associated with each neuron is a context function c: 𝒵 →𝒞, where 𝒵 is the set of possible side information and 𝒞={0, …, k-1} for some k ∈ℕ is the context space. Each neuron i is now parameterized by a weight matrix Wᵢ=[w_(i, 0) …w_(i, k-1)]^⊤ with each row vector w_(i j) ∈𝒲 for 0 ≤j<k. The context function c is responsible for mapping side information z ∈𝒵 to a particular row w_(i, c(z)) of Wᵢ, which we then use to weight the Product of Gaussians. In other words, a G-GLN neuron can be defined by:

PoG_Wᶜ(y ; f₁(·), …, fₘ(·), z):=PoG_(wᶜ⁽ᶻ⁾)(y ; f₁(·), …, fₘ(·))

with the associated loss function -log(PoG_Wᶜ(y ; f₁(y), …, fₘ(y), z)) inheriting all the properties needed to apply Online Convex Programming.