PCFG SET (Probabilistic Context Free Grammar String Edit Task)

The Probabilistic Context Free Grammar String Edit Task (PCFG SET) dataset is a dataset with sequence to sequence problems specifically designed to test different aspects of compositional generalisation. In particular, the dataset contains splits to test for systematicity, productivity, substitutivity, localism and overgeneralisation.

The input alphabet of PCFG SET contains three types of words: words for unary and binary functions that represent \emph{string edit operations} (e.g. append, copy, reverse), elements to form the string sequences that these functions can be applied to (e.g. A, B, A1, B1), and a separator to separate the arguments of a binary function (,). The input sequences that are formed with this alphabet are sequences describing how a series of such operations are to be applied to a string argument. For instance:

• repeat A B C
• echo remove_first D K , E F
• append swap F G H , repeat I J

The input sequences are generated with a PCFG, whose production probabilities are learned with EM to match the depth and length distributions in a corpus with English sentences.

The output of a PCFG SET sequence, representing its meaning, is constructed by recursively applying the string edit operations specified in the sequence. For instance:

• repeat A B C & → & A B C A B C
• echo remove_first D K , E F & → & E F F
• append swap F G H , repeat I J & → & H G F I J I J

The string alphabet used for the construction of the dataset has 520 distinct elements, the length of the string arguments to a functions is limited to 5.The dataset contains around 100 thousand examples in total. A full description of the dataset can be found in Hupkes et al (2020).