{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/temporal-modulation-network-for-controllable","title":"Temporal Modulation Network for Controllable Space-Time Video Super-Resolution","arxiv_id":"2104.10642","date":"2021-04-21","proceeding":"CVPR 2021 1","authors":["Gang Xu","Jun Xu","Zhen Li","Liang Wang","Xing Sun","Ming-Ming Cheng"],"abstract":"Space-time video super-resolution (STVSR) aims to increase the spatial and temporal resolutions of low-resolution and low-frame-rate videos. Recently, deformable convolution based methods have achieved promising STVSR performance, but they could only infer the intermediate frame pre-defined in the training stage. Besides, these methods undervalued the short-term motion cues among adjacent frames. In this paper, we propose a Temporal Modulation Network (TMNet) to interpolate arbitrary intermediate frame(s) with accurate high-resolution reconstruction. Specifically, we propose a Temporal Modulation Block (TMB) to modulate deformable convolution kernels for controllable feature interpolation. To well exploit the temporal information, we propose a Locally-temporal Feature Comparison (LFC) module, along with the Bi-directional Deformable ConvLSTM, to extract short-term and long-term motion cues in videos. Experiments on three benchmark datasets demonstrate that our TMNet outperforms previous STVSR methods. The code is available at https://github.com/CS-GangXu/TMNet.","url_abs":"https://arxiv.org/abs/2104.10642v2","url_pdf":"https://arxiv.org/pdf/2104.10642v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"temporal-modulation-network-for-controllable","repo_url":"https://github.com/CS-GangXu/TMNet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"space-time-video-super-resolution","task_name":"Space-time Video Super-resolution"},{"task_slug":"super-resolution","task_name":"Super-Resolution"},{"task_slug":"video-super-resolution","task_name":"Video Super-Resolution"}],"methods":[{"method_slug":"convlstm","method_name":"ConvLSTM"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"deformable-convolution","method_name":"Deformable Convolution"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-super-resolution-on-msu-super-1","task":"Video Super-Resolution","dataset":"MSU Super-Resolution for Video Compression","model":"TMNet + x264","rank_in_archive_order":50,"of":85,"metrics":{"BSQ-rate over ERQA":"1.879","BSQ-rate over LPIPS":"1.377","BSQ-rate over MS-SSIM":"0.844","BSQ-rate over PSNR":"1.481","BSQ-rate over VMAF":"1.061"},"uses_additional_data":false},{"leaderboard":"/sota/video-super-resolution-on-msu-super-1","task":"Video Super-Resolution","dataset":"MSU Super-Resolution for Video Compression","model":"TMNet + uavs3e","rank_in_archive_order":66,"of":85,"metrics":{"BSQ-rate over ERQA":"13.187","BSQ-rate over LPIPS":"5.015","BSQ-rate over MS-SSIM":"4.317","BSQ-rate over PSNR":"15.144","BSQ-rate over VMAF":"3.487"},"uses_additional_data":false},{"leaderboard":"/sota/video-super-resolution-on-msu-super-1","task":"Video Super-Resolution","dataset":"MSU Super-Resolution for Video Compression","model":"TMNet + x265","rank_in_archive_order":70,"of":85,"metrics":{"BSQ-rate over ERQA":"13.577","BSQ-rate over LPIPS":"13.485","BSQ-rate over MS-SSIM":"1.735","BSQ-rate over PSNR":"7.046","BSQ-rate over VMAF":"2.009"},"uses_additional_data":false},{"leaderboard":"/sota/video-super-resolution-on-msu-super-1","task":"Video Super-Resolution","dataset":"MSU Super-Resolution for Video Compression","model":"TMNet + vvenc","rank_in_archive_order":80,"of":85,"metrics":{"BSQ-rate over ERQA":"21.303","BSQ-rate over LPIPS":"13.988","BSQ-rate over MS-SSIM":"1.813","BSQ-rate over PSNR":"9.43","BSQ-rate over VMAF":"1.795"},"uses_additional_data":false},{"leaderboard":"/sota/video-super-resolution-on-msu-super-1","task":"Video Super-Resolution","dataset":"MSU Super-Resolution for Video Compression","model":"TMNet + aomenc","rank_in_archive_order":81,"of":85,"metrics":{"BSQ-rate over ERQA":"21.798","BSQ-rate over LPIPS":"6.276","BSQ-rate over MS-SSIM":"10.322","BSQ-rate over PSNR":"15.144","BSQ-rate over VMAF":"4.667"},"uses_additional_data":false},{"leaderboard":"/sota/video-super-resolution-on-msu-vsr-benchmark","task":"Video Super-Resolution","dataset":"MSU Video Super Resolution Benchmark: Detail Restoration","model":"TMNet","rank_in_archive_order":8,"of":32,"metrics":{"1 - LPIPS":"0.931","ERQAv1.0":"0.712","FPS":"1.136","PSNR":"30.364","QRCRv1.0":"0.549","SSIM":"0.885","Subjective score":"6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.10642","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.10642"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/CS-GangXu/TMNet","reach":null}],"summary":{"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1c814131165d7ef0","entry":"TMB","repo":"CS-GangXu/TMNet","repo_kind":"official","path":"models/modules/STVSR.py","file_url":"https://github.com/CS-GangXu/TMNet/blob/HEAD/models/modules/STVSR.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1c814131165d7ef0"}},{"code_sha256_prefix":"43ed322f20337c3b","entry":"BiDeformableConvLSTM","repo":"CS-GangXu/TMNet","repo_kind":"official","path":"models/modules/STVSR.py","file_url":"https://github.com/CS-GangXu/TMNet/blob/HEAD/models/modules/STVSR.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"43ed322f20337c3b"}},{"code_sha256_prefix":"8cdc0946aaa23d7a","entry":"DeformableConvLSTM","repo":"CS-GangXu/TMNet","repo_kind":"official","path":"models/modules/STVSR.py","file_url":"https://github.com/CS-GangXu/TMNet/blob/HEAD/models/modules/STVSR.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8cdc0946aaa23d7a"}},{"code_sha256_prefix":"e4122e785ab49435","entry":"Easy_PCD","repo":"CS-GangXu/TMNet","repo_kind":"official","path":"models/modules/STVSR.py","file_url":"https://github.com/CS-GangXu/TMNet/blob/HEAD/models/modules/STVSR.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4122e785ab49435"}},{"code_sha256_prefix":"f6dd4f92cf1e6361","entry":"PCD_Align","repo":"CS-GangXu/TMNet","repo_kind":"official","path":"models/modules/STVSR.py","file_url":"https://github.com/CS-GangXu/TMNet/blob/HEAD/models/modules/STVSR.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6dd4f92cf1e6361"}},{"code_sha256_prefix":"1dec0a72229da15f","entry":"TMNet","repo":"CS-GangXu/TMNet","repo_kind":"official","path":"models/modules/STVSR.py","file_url":"https://github.com/CS-GangXu/TMNet/blob/HEAD/models/modules/STVSR.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1dec0a72229da15f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}