Papers › Weakly-supervised learning of visual relations
Weakly-supervised learning of visual relations
Julia Peyre, Ivan Laptev, Cordelia Schmid, Josef Sivic
This paper introduces a novel approach for modeling visual relations between pairs of objects. We call relation a triplet of the form (subject, predicate, object) where the predicate is typically a preposition (eg. 'under', 'in front of') or a verb ('hold', 'ride') that links a pair of objects (subject, object). Learning such relations is challenging as the objects have different spatial configurations and appearances depending on the relation in which they occur. Another major challenge comes from the difficulty to get annotations, especially at box-level, for all possible triplets, which makes both learning and evaluation difficult. The contributions of this paper are threefold. First, we design strong yet flexible visual features that encode the appearance and spatial configuration for pairs of objects. Second, we propose a weakly-supervised discriminative clustering model to learn relations from image-level labels only. Third we introduce a new challenging dataset of unusual relations (UnRel) together with an exhaustive annotation, that enables accurate evaluation of visual relation retrieval. We show experimentally that our model results in state-of-the-art results on the visual relationship dataset significantly improving performance on previously unseen relations (zero-shot learning), and confirm this observation on our newly introduced UnRel dataset.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
2 archive task tags without a task page not shown.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Visual Relationship Detection | VRD Phrase Detection | Peyre et. al [[Peyre et al.2017]] | R@100 | 19.5 | #6 of 7 | Archive leaderboard | report |
| Visual Relationship Detection | VRD Phrase Detection | Peyre et. al [[Peyre et al.2017]] | R@50 | 17.9 | #6 of 7 | Archive leaderboard | report |
| Visual Relationship Detection | VRD Predicate Detection | Peyre et. al [[Peyre et al.2017]] | R@100 | 52.6 | #5 of 7 | Archive leaderboard | report |
| Visual Relationship Detection | VRD Predicate Detection | Peyre et. al [[Peyre et al.2017]] | R@50 | 52.6 | #5 of 7 | Archive leaderboard | report |
| Visual Relationship Detection | VRD Relationship Detection | Peyre et. al [[Peyre et al.2017]] | R@100 | 17.1 | #6 of 8 | Archive leaderboard | report |
| Visual Relationship Detection | VRD Relationship Detection | Peyre et. al [[Peyre et al.2017]] | R@50 | 15.8 | #6 of 8 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections