Papers › Universal Instance Perception as Object Discovery and Retrieval

Universal Instance Perception as Object Discovery and Retrieval

12 Mar 2023CVPR 2023 1arXiv:2303.06674archive 2025-07-28

Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, Huchuan Lu

All instance perception tasks aim at finding certain objects specified by some queries such as category names, language expressions, and target annotations, but this complete field has been split into multiple independent subtasks. In this work, we present a universal instance perception model of the next generation, termed UNINEXT. UNINEXT reformulates diverse instance perception tasks into a unified object discovery and retrieval paradigm and can flexibly perceive different types of objects by simply changing the input prompts. This unified formulation brings the following benefits: (1) enormous data from different tasks and label vocabularies can be exploited for jointly training general instance-level representations, which is especially beneficial for tasks lacking in training data. (2) the unified model is parameter-efficient and can save redundant computation when handling multiple tasks simultaneously. UNINEXT shows superior performance on 20 challenging benchmarks from 10 instance-level tasks including classical image-level tasks (object detection and instance segmentation), vision-and-language tasks (referring expression comprehension and segmentation), and six video-level object tracking tasks. Code is available at https://github.com/MasterBin-IIAU/UNINEXT.

PaperPDFConference PDFCodeCode Syntology ran

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, Syntology's MCP tool lists every function and class Syntology harvested from this paper and whether it ran (how to connect): get_harvested_code_for_paper(arxiv_id="2303.06674")

Code

Syntology Ran 3 of 4 code samples harvested from 1 repository linked to this paper; 1 has no recorded run. Of those that ran: 1 ran · honoured contract; 1 ran · our draft was wrong; 1 ran with no contract checked.

By repository: official repository: 4 samples from 1 repository, 3 ran. The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

MasterBin-IIAU/UNINEXT officialmentioned in papermentioned on GitHubpytorchMIT report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

4 samples harvested; 3 ran; 1 honoured the contract we drafted; 1 has no recorded run. Read from Syntology's graph 2026-09-24; that is when this build read the record, not when the samples ran.

1ran · honoured contract
1ran · our draft was wrong
1ran
1unverified

Licence: 0 of the 4 samples are pointer only, meaning Syntology does not serve that copy's text. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from MasterBin-IIAU/UNINEXT. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself; “our draft was wrong” and “fixture could not drive it” are failures of Syntology's instrument, not of the code.

Each sample ends with its code_sha256, Syntology's identity for that exact code. An agent fetches the stored sample with Syntology's MCP tool get_code(code_sha256="…") (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples from a repository marked official are listed first. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of four reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; or the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar). Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

aligned_bilinear MasterBin-IIAU/UNINEXT/projects/UNINEXT/uninext/models/ddetrs.py official repository ran · our draft was wrong fingerprinted MIT (permissive) · 29dbbaafa4be18e1 · report
compute_locations MasterBin-IIAU/UNINEXT/projects/UNINEXT/uninext/models/ddetrs.py official repository ran · honoured contract MIT (permissive) · ed1a308adb65f645 · report
parse_dynamic_params MasterBin-IIAU/UNINEXT/projects/UNINEXT/uninext/models/ddetrs.py official repository ran MIT (permissive) · 46f34dc9f9a94b67 · report
make_full_size MasterBin-IIAU/UNINEXT/uninext_vots.py official repository unverified MIT (permissive) · 8edb076e926c9e81 · report

Tasks

Described Object DetectionGeneralized Referring Expression ComprehensionInstance SegmentationMulti-Object Tracking and SegmentationMultiple Object TrackingObjectObject DetectionObject DiscoveryObject TrackingReferring ExpressionReferring Expression ComprehensionReferring Expression SegmentationReferring Video Object SegmentationRetrievalSemantic SegmentationVideo Instance SegmentationVisual Object TrackingVisual TrackingZero Shot Segmentationobject-detection

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Described Object Detection Description Detection Dataset UNINEXT-large Intra-scenario ABS mAP 15.9 #5 of 8 Archive leaderboard report
Described Object Detection Description Detection Dataset UNINEXT-large Intra-scenario FULL mAP 17.9 #5 of 8 Archive leaderboard report
Described Object Detection Description Detection Dataset UNINEXT-large Intra-scenario PRES mAP 18.6 #5 of 8 Archive leaderboard report
Generalized Referring Expression Comprehension gRefCOCO UNINEXT N-acc. 50.6 #2 of 5 Archive leaderboard report
Generalized Referring Expression Comprehension gRefCOCO UNINEXT Precision@(F1=1, IoU≥0.5) 58.2 #2 of 5 Archive leaderboard report
Instance Segmentation COCO test-dev UNINEXT-H AP50 76.2 #17 of 112 Archive leaderboard report
Instance Segmentation COCO test-dev UNINEXT-H AP75 56.7 #17 of 112 Archive leaderboard report
Instance Segmentation COCO test-dev UNINEXT-H APL 67.5 #17 of 112 Archive leaderboard report
Instance Segmentation COCO test-dev UNINEXT-H APM 55.9 #17 of 112 Archive leaderboard report
Instance Segmentation COCO test-dev UNINEXT-H APS 33.3 #17 of 112 Archive leaderboard report
Instance Segmentation COCO test-dev UNINEXT-H mask AP 51.8 #17 of 112 Archive leaderboard report
Multi-Object Tracking and Segmentation BDD100K val UNINEXT-H mMOTSA 35.7 #1 of 8 Archive leaderboard report
Multiple Object Tracking BDD100K val UNINEXT-H AssocA - #2 of 9 Archive leaderboard report
Multiple Object Tracking BDD100K val UNINEXT-H TETA - #2 of 9 Archive leaderboard report
Multiple Object Tracking BDD100K val UNINEXT-H mIDF1 56.7 #2 of 9 Archive leaderboard report
Multiple Object Tracking BDD100K val UNINEXT-H mMOTA 44.2 #2 of 9 Archive leaderboard report
Object Detection COCO minival UNINEXT-H AP50 77.5 #21 of 220 Archive leaderboard report
Object Detection COCO minival UNINEXT-H AP75 66.7 #21 of 220 Archive leaderboard report
Object Detection COCO minival UNINEXT-H APL 75.3 #21 of 220 Archive leaderboard report
Object Detection COCO minival UNINEXT-H APM 64.8 #21 of 220 Archive leaderboard report
Object Detection COCO minival UNINEXT-H APS 45.1 #21 of 220 Archive leaderboard report
Object Detection COCO minival UNINEXT-H box AP 60.6 #21 of 220 Archive leaderboard report
Referring Expression Segmentation DAVIS 2017 (val) UNINEXT-H J&F 1st frame 72.5 #1 of 18 Archive leaderboard report
Referring Expression Segmentation RefCOCO+ test B UNINEXT-H Overall IoU 66.22 #8 of 30 Archive leaderboard report
Referring Expression Segmentation RefCOCO+ testA UNINEXT-H Overall IoU 76.42 #9 of 30 Archive leaderboard report
Referring Expression Segmentation RefCOCO+ val UNINEXT-H Overall IoU 72.47 #11 of 33 Archive leaderboard report
Referring Expression Segmentation RefCoCo val UNINEXT-H Overall IoU 82.19 #7 of 37 Archive leaderboard report
Referring Expression Segmentation Refer-YouTube-VOS (2021 public validation) UNINEXT-H F 72.7 #4 of 33 Archive leaderboard report
Referring Expression Segmentation Refer-YouTube-VOS (2021 public validation) UNINEXT-H J 67.6 #4 of 33 Archive leaderboard report
Referring Expression Segmentation Refer-YouTube-VOS (2021 public validation) UNINEXT-H J&F 70.1 #4 of 33 Archive leaderboard report
Video Instance Segmentation OVIS validation UNINEXT (ViT-H, Online) AP50 72.5 #7 of 44 Archive leaderboard report
Video Instance Segmentation OVIS validation UNINEXT (ViT-H, Online) AP75 52.2 #7 of 44 Archive leaderboard report
Video Instance Segmentation OVIS validation UNINEXT (ViT-H, Online) mask AP 49.0 #7 of 44 Archive leaderboard report
Video Instance Segmentation OVIS validation UNINEXT (ResNet-50, Online) AP50 55.5 #26 of 44 Archive leaderboard report
Video Instance Segmentation OVIS validation UNINEXT (ResNet-50, Online) AP75 35.6 #26 of 44 Archive leaderboard report
Video Instance Segmentation OVIS validation UNINEXT (ResNet-50, Online) mask AP 34.0 #26 of 44 Archive leaderboard report
Visual Object Tracking LaSOT UNINEXT-L AUC 72.4 #19 of 46 Archive leaderboard report
Visual Object Tracking LaSOT UNINEXT-L Normalized Precision 80.7 #19 of 46 Archive leaderboard report
Visual Object Tracking LaSOT UNINEXT-L Precision 78.9 #19 of 46 Archive leaderboard report
Visual Object Tracking LaSOT UNINEXT-H AUC 72.2 #21 of 46 Archive leaderboard report
Visual Object Tracking LaSOT UNINEXT-H Normalized Precision 80.8 #21 of 46 Archive leaderboard report
Visual Object Tracking LaSOT UNINEXT-H Precision 79.4 #21 of 46 Archive leaderboard report
Visual Object Tracking LaSOT-ext UNINEXT-H AUC 56.2 #5 of 18 Archive leaderboard report
Visual Object Tracking LaSOT-ext UNINEXT-H Normalized Precision 63.8 #5 of 18 Archive leaderboard report
Visual Object Tracking LaSOT-ext UNINEXT-H Precision 63.8 #5 of 18 Archive leaderboard report
Visual Object Tracking TrackingNet UNINEXT-H Accuracy 85.4 #13 of 40 Archive leaderboard report
Visual Object Tracking TrackingNet UNINEXT-H Normalized Precision 89.0 #13 of 40 Archive leaderboard report
Visual Object Tracking TrackingNet UNINEXT-H Precision 86.4 #13 of 40 Archive leaderboard report
Visual Tracking TNL2K UNINEXT-H AUC 59.3 #2 of 6 Archive leaderboard report
Visual Tracking TNL2K UNINEXT-H precision 62.8 #2 of 6 Archive leaderboard report
Zero Shot Segmentation Segmentation in the Wild UNINEXT Mean AP 42.1 #3 of 12 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections