SKEP

SKEP is a self-supervised pre-training method for sentiment analysis. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair.

SKEP contains two parts: (1) Sentiment masking recognizes the sentiment information of an input sequence based on automatically-mined sentiment knowledge, and produces a corrupted version by removing these informations. (2) Sentiment pre-training objectives require the transformer to recover the removed information from the corrupted version. The three prediction objectives on top are jointly optimized: Sentiment Word (SW) prediction (on .x₉), Word Polarity (SP) prediction (on x₆ and 𝐱₉ ), Aspect-Sentiment pairs (AP) prediction (on 𝐱₁ ). Here, the smiley denotes positive polarity. Notably, on x₆, only SP is calculated without SW, as its original word has been predicted in the pair prediction on 𝐱₁.