Papers › Order Matters: Sequence to sequence for sets

Order Matters: Sequence to sequence for sets

19 Nov 2015arXiv:1511.06391archive 2025-07-28

Oriol Vinyals, Samy Bengio, Manjunath Kudlur

Sequences have become first class citizens in supervised learning thanks to the resurgence of recurrent neural networks. Many complex tasks that require mapping from or to a sequence of observations can now be formulated with the sequence-to-sequence (seq2seq) framework which employs the chain rule to efficiently represent the joint probability of sequences. In many cases, however, variable sized inputs and/or outputs might not be naturally expressed as sequences. For instance, it is not clear how to input a set of numbers into a model where the task is to sort them; similarly, we do not know how to organize outputs when they correspond to random variables and the task is to model their unknown joint probability. In this paper, we first show using various examples that the order in which we organize input and/or output data matters significantly when learning an underlying model. We then discuss an extension of the seq2seq framework that goes beyond sequences and handles input sets in a principled way. In addition, we propose a loss which, by searching over possible orders during training, deals with the lack of structure of output sets. We show empirical evidence of our claims regarding ordering, and on the modifications to the seq2seq framework on benchmark language modeling and parsing tasks, as well as two artificial tasks -- sorting numbers and estimating the joint probability of unknown graphical models.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

arunppsg/set2set mentioned on GitHubpytorchMITSyntology: not harvested report
davidtangGT/MEGNET mentioned on GitHubtfBSD-3-ClauseSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Relational inductive biases, deep learning, and graph networks (arXiv:1806.01261; that paper's own run record: 39 ran (of which 5 constructed an object rather than computing a result; 38 with no instrument failure: 4 honoured, 0 violated, 34 with no contract checked; 1 where Syntology's instrument failed) · 12 unverified); nothing checks that they implement this paper's method. report
materialsvirtuallab/megnet mentioned on GitHubtfBSD-3-ClauseSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals (arXiv:1812.05055; that paper's own run record: 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
pyg-team/pytorch_geometric mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Learning the Geometry of Data: A Mathematical Review of Shape Space Analysis (arXiv:2606.17022; that paper's own run record: 8 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified); nothing checks that they implement this paper's method. report
teddykoker/mpnn-for-quantum-chem mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Neural Message Passing for Quantum Chemistry (arXiv:1704.01212; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
threelittlemonkeys/pointer-network-pytorch mentioned on GitHubpytorchSyntology: not harvested report
MindSpore-paper-code-2/code400 mindsporeSyntology: not harvested report
dmlc/dgl pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to node2vec: Scalable Feature Learning for Networks (arXiv:1607.00653; that paper's own run record: 15 ran (of which 2 constructed an object rather than computing a result; 12 with no instrument failure: 2 honoured, 0 violated, 10 with no contract checked; 3 where Syntology's instrument failed) · 10 unverified), Interaction Networks for Learning about Objects, Relations and Physics (arXiv:1612.00222; that paper's own run record: 6 ran (of which 6 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified; every one of the 6 samples that ran constructed an object rather than computing a result), Junction Tree Variational Autoencoder for Molecular Graph Generation (arXiv:1802.04364; that paper's own run record: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result) and 3 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
pyg-team/pytorch_geometric pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Learning the Geometry of Data: A Mathematical Review of Shape Space Analysis (arXiv:2606.17022; that paper's own run record: 8 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Language Modeling

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

LSTMSeq2SeqSigmoid ActivationTanh Activation

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections