Papers › Recurrent Scene Parsing with Perspective Understanding in the Loop

Recurrent Scene Parsing with Perspective Understanding in the Loop

20 May 2017CVPR 2018 6arXiv:1705.07238archive 2025-07-28

Shu Kong, Charless Fowlkes

Objects may appear at arbitrary scales in perspective images of a scene, posing a challenge for recognition systems that process images at a fixed resolution. We propose a depth-aware gating module that adaptively selects the pooling field size in a convolutional network architecture according to the object scale (inversely proportional to the depth) so that small details are preserved for distant objects while larger receptive fields are used for those nearby. The depth gating signal is provided by stereo disparity or estimated directly from monocular input. We integrate this depth-aware gating into a recurrent convolutional neural network to perform semantic segmentation. Our recurrent module iteratively refines the segmentation results, leveraging the depth and semantic predictions from the previous iterations. Through extensive experiments on four popular large-scale RGB-D datasets, we demonstrate this approach achieves competitive semantic segmentation performance with a model which is substantially more compact. We carry out extensive analysis of this architecture including variants that operate on monocular RGB but use depth as side-information during training, unsupervised gating as a generic attentional mechanism, and multi-resolution gating. We find that gated pooling for joint semantic segmentation and depth yields state-of-the-art results for quantitative monocular depth estimation.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

aimerykong/Recurrent-Scene-Parsing-with-Perspective-Understanding-in-the-loop officialmentioned in paper report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Depth Estimation Monocular Depth Estimation Scene Parsing Segmentation Semantic Segmentation

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Semantic Segmentation	Cityscapes test	DepthSeg (ResNet-101)	Mean IoU (class)	78.2%	#61 of 105	Archive leaderboard	report
Semantic Segmentation	NYU Depth v2	RecurrentSceneParsing	Mean IoU	44.5%	#96 of 121	Archive leaderboard	report
Semantic Segmentation	SUN-RGBD	DPLNet	Mean IoU	45.1%	#40 of 44	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

1x1 Convolution Average Pooling Batch Normalization Bottleneck Residual Block Convolution Global Average Pooling Kaiming Initialization Max Pooling ReLU Residual Block Residual Connection

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections