Papers › RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution

RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution

27 Mar 2022CVPR 2022 1arXiv:2203.14186archive 2025-07-28

Zhicheng Geng, Luming Liang, Tianyu Ding, Ilya Zharkov

Space-time video super-resolution (STVSR) is the task of interpolating videos with both Low Frame Rate (LFR) and Low Resolution (LR) to produce High-Frame-Rate (HFR) and also High-Resolution (HR) counterparts. The existing methods based on Convolutional Neural Network~(CNN) succeed in achieving visually satisfied results while suffer from slow inference speed due to their heavy architectures. We propose to resolve this issue by using a spatial-temporal transformer that naturally incorporates the spatial and temporal super resolution modules into a single model. Unlike CNN-based methods, we do not explicitly use separated building blocks for temporal interpolations and spatial super-resolutions; instead, we only use a single end-to-end transformer architecture. Specifically, a reusable dictionary is built by encoders based on the input LFR and LR frames, which is then utilized in the decoder part to synthesize the HFR and HR frames. Compared with the state-of-the-art TMNet \cite{xu2021temporal}, our network is 60% smaller (4.5M vs 12.3M parameters) and 80% faster (26.2fps vs 14.3fps on 720×576 frames) without sacrificing much performance. The source code is available at https://github.com/llmpass/RSTT.

PaperPDFConference PDFCodeCode Syntology ran

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, Syntology's MCP tool lists every function and class Syntology harvested from this paper and whether it ran (how to connect): get_harvested_code_for_paper(arxiv_id="2203.14186")

Code

Syntology Ran 8 of 14 code samples harvested from 1 repository linked to this paper; 6 have no recorded run. Of those that ran: 1 ran · our draft was wrong; 2 ran · fixture could not drive it; 5 ran with no contract checked.

By repository: official repository: 14 samples from 1 repository, 8 ran. The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

llmpass/RSTT officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

14 samples harvested; 8 ran; 0 honoured the contract we drafted; 6 have no recorded run. Read from Syntology's graph 2026-09-24; that is when this build read the record, not when the samples ran.

1ran · our draft was wrong
2ran · fixture could not drive it
5ran
6unverified

Licence: 14 of the 14 samples are pointer only, meaning Syntology does not serve that copy's text. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from llmpass/RSTT. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself; “our draft was wrong” and “fixture could not drive it” are failures of Syntology's instrument, not of the code.

Each sample ends with its code_sha256, Syntology's identity for that exact code. An agent fetches the stored sample with Syntology's MCP tool get_code(code_sha256="…") (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples from a repository marked official are listed first. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of four reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; or the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar). Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

Downsample llmpass/RSTT/models/RSTT.py official repository ran no licence file found · pointer only · 190b1325503c4ab8 · report
InputProj llmpass/RSTT/models/RSTT.py official repository ran no licence file found · pointer only · 3aee7fafe363d322 · report
ResidualBlock_noBN llmpass/RSTT/models/RSTT.py official repository ran no licence file found · pointer only · b80f785c8fa3697e · report
Upsample llmpass/RSTT/models/RSTT.py official repository ran no licence file found · pointer only · 9d6b9fecff821851 · report
WindowAttention3D llmpass/RSTT/models/RSTT.py official repository ran no licence file found · pointer only · 6f69c55b59c9744c · report
get_window_size llmpass/RSTT/models/RSTT.py official repository ran · fixture could not drive it no licence file found · pointer only · d306f10d62b6ae0c · report
window_partition llmpass/RSTT/models/RSTT.py official repository ran · fixture could not drive it no licence file found · pointer only · dbf81aece2a63747 · report
window_reverse llmpass/RSTT/models/RSTT.py official repository ran · our draft was wrong no licence file found · pointer only · 15e141e09e3e5d8d · report
DecoderLayer llmpass/RSTT/models/RSTT.py official repository unverified no licence file found · pointer only · cb3cd3a0e1663f15 · report
EncoderLayer llmpass/RSTT/models/RSTT.py official repository unverified no licence file found · pointer only · 8e283aa9300b35c3 · report
RSTT llmpass/RSTT/models/RSTT.py official repository unverified no licence file found · pointer only · 6c382b558756a30b · report
VSTSRDecoderTransformerBlock llmpass/RSTT/models/RSTT.py official repository unverified no licence file found · pointer only · 5b1fddb874b78287 · report
VSTSREncoderTransformerBlock llmpass/RSTT/models/RSTT.py official repository unverified no licence file found · pointer only · 2206f8156a0c0bc6 · report
initialize_weights llmpass/RSTT/models/RSTT.py official repository unverified no licence file found · pointer only · e217d1b65eaca404 · report

Tasks

DecoderSpace-time Video Super-resolutionSuper-ResolutionVideo Frame InterpolationVideo Super-Resolution

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Space-time Video Super-resolution Vimeo90K-Fast RSTT-L PSNR 36.80 #2 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Fast RSTT-L SSIM 0.9403 #2 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Fast RSTT-M PSNR 36.78 #3 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Fast RSTT-M SSIM 0.9401 #3 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Fast RSTT-S PSNR 36.58 #4 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Fast RSTT-S SSIM 0.9381 #4 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Medium RSTT-L PSNR 35.66 #2 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Medium RSTT-L SSIM 0.9381 #2 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Medium RSTT-M PSNR 35.62 #3 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Medium RSTT-M SSIM 0.9377 #3 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Medium RSTT-S PSNR 35.43 #4 of 4 Archive leaderboard report
Space-time Video Super-resolution Vimeo90K-Medium RSTT-S SSIM 0.9358 #4 of 4 Archive leaderboard report
Video Frame Interpolation Vid4 - 4x upscaling RSTT-L PSNR 26.43 #2 of 5 Archive leaderboard report
Video Frame Interpolation Vid4 - 4x upscaling RSTT-L Parameters 7670000 #2 of 5 Archive leaderboard report
Video Frame Interpolation Vid4 - 4x upscaling RSTT-L SSIM 0.7994 #2 of 5 Archive leaderboard report
Video Frame Interpolation Vid4 - 4x upscaling RSTT-M PSNR 26.37 #3 of 5 Archive leaderboard report
Video Frame Interpolation Vid4 - 4x upscaling RSTT-M Parameters 6080000 #3 of 5 Archive leaderboard report
Video Frame Interpolation Vid4 - 4x upscaling RSTT-M SSIM 0.7978 #3 of 5 Archive leaderboard report
Video Frame Interpolation Vid4 - 4x upscaling RSTT-S PSNR 26.29 #5 of 5 Archive leaderboard report
Video Frame Interpolation Vid4 - 4x upscaling RSTT-S Parameters 4490000 #5 of 5 Archive leaderboard report
Video Frame Interpolation Vid4 - 4x upscaling RSTT-S SSIM 0.7941 #5 of 5 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

AttentionDense ConnectionsLayer NormalizationLinear LayerMulti-Head AttentionResidual ConnectionSPEEDSoftmaxStochastic DepthSwin Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections