SimAdapter

SimAdapter is a module for explicitly learning knowledge from adapters. SimAdapter aims to learn the similarities between the source and target languages during fine-tuning using the adapters, and the similarity is based on an attention mechanism.

The detailed composition of the SimAdapter is shown in the Figure. By taking the language-agnostic representations from the backbone model as the query, and the language-specific outputs from multiple adapter as the keys and values, the final output for SimAdapter over attention are computed as (For notation simplicity, we omit the layer index l below):

SimAdapter(𝐳, 𝐚_(S₁, S₂, …, S_N))=∑ᵢ₌₁ᴺ Attn(𝐳, 𝐚_(Sᵢ)) ·(𝐚_(Sᵢ) 𝐖_V)

where SimAdapter (·) and Attn(·) denotes the SimAdapter and attention operations, respectively. Specifically, the attention operation is computed as:

Attn(𝐳, 𝐚)=Softmax(((𝐳 𝐖_Q)(𝐚 𝐖_K)^⊤)/τ)

where τ is the temperature coefficient, 𝐖_Q, 𝐖_K, 𝐖_V are attention matrices. Note that while 𝐖_Q, 𝐖_K are initialized randomly, 𝐖_V is initialized with a diagonal of ones and the rest of the matrix with small weights (1 e-6) to retain the adapter representations. Furthermore, a regularization term is introduced to avoid drastic feature changes:

ℒ_(reg)=∑_(i, j)((𝐈_V)_(i, j)-(𝐖_V)_(i, j))²

where 𝐈_V is the identity matrix with the same size as 𝐖_V