{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-trajectory-aware-transformer-for","title":"Learning Trajectory-Aware Transformer for Video Super-Resolution","arxiv_id":"2204.04216","date":"2022-04-08","proceeding":"CVPR 2022 1","authors":["Chengxu Liu","Huan Yang","Jianlong Fu","Xueming Qian"],"abstract":"Video super-resolution (VSR) aims to restore a sequence of high-resolution (HR) frames from their low-resolution (LR) counterparts. Although some progress has been made, there are grand challenges to effectively utilize temporal dependency in entire video sequences. Existing approaches usually align and aggregate video frames from limited adjacent frames (e.g., 5 or 7 frames), which prevents these approaches from satisfactory results. In this paper, we take one step further to enable effective spatio-temporal learning in videos. We propose a novel Trajectory-aware Transformer for Video Super-Resolution (TTVSR). In particular, we formulate video frames into several pre-aligned trajectories which consist of continuous visual tokens. For a query token, self-attention is only learned on relevant visual tokens along spatio-temporal trajectories. Compared with vanilla vision Transformers, such a design significantly reduces the computational cost and enables Transformers to model long-range features. We further propose a cross-scale feature tokenization module to overcome scale-changing problems that often occur in long-range videos. Experimental results demonstrate the superiority of the proposed TTVSR over state-of-the-art models, by extensive quantitative and qualitative evaluations in four widely-used video super-resolution benchmarks. Both code and pre-trained models can be downloaded at https://github.com/researchmm/TTVSR.","url_abs":"https://arxiv.org/abs/2204.04216v3","url_pdf":"https://arxiv.org/pdf/2204.04216v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-trajectory-aware-transformer-for","repo_url":"https://github.com/researchmm/TTVSR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"super-resolution","task_name":"Super-Resolution"},{"task_slug":"video-super-resolution","task_name":"Video Super-Resolution"},{"task_slug":"video-deraining","task_name":"Video deraining"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-super-resolution-on-udm10-4x-upscaling","task":"Video Super-Resolution","dataset":"UDM10 - 4x upscaling","model":"TTVSR","rank_in_archive_order":4,"of":7,"metrics":{"PSNR":"40.41","SSIM":"0.9712"},"uses_additional_data":false},{"leaderboard":"/sota/video-super-resolution-on-vid4-4x-upscaling-1","task":"Video Super-Resolution","dataset":"Vid4 - 4x upscaling - BD degradation","model":"TTVSR","rank_in_archive_order":6,"of":18,"metrics":{"PSNR":"28.40","SSIM":"0.8643"},"uses_additional_data":false},{"leaderboard":"/sota/video-deraining-on-vrds","task":"Video deraining","dataset":"VRDS","model":"TTVSR","rank_in_archive_order":6,"of":8,"metrics":{"PSNR":"28.05","SSIM":"0.8998"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.04216","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.04216"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/researchmm/TTVSR","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d538533133adaba1","entry":"bbox2mask","repo":"researchmm/TTVSR","repo_kind":"official","path":"mmedit/core/mask.py","file_url":"https://github.com/researchmm/TTVSR/blob/HEAD/mmedit/core/mask.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d538533133adaba1"}},{"code_sha256_prefix":"a728ca5bee3c4984","entry":"brush_stroke_mask","repo":"researchmm/TTVSR","repo_kind":"official","path":"mmedit/core/mask.py","file_url":"https://github.com/researchmm/TTVSR/blob/HEAD/mmedit/core/mask.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a728ca5bee3c4984"}},{"code_sha256_prefix":"da019d97d47ae83c","entry":"dgaussian","repo":"researchmm/TTVSR","repo_kind":"official","path":"mmedit/core/evaluation/metric_utils.py","file_url":"https://github.com/researchmm/TTVSR/blob/HEAD/mmedit/core/evaluation/metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da019d97d47ae83c"}},{"code_sha256_prefix":"6140a018695b3869","entry":"gauss_filter","repo":"researchmm/TTVSR","repo_kind":"official","path":"mmedit/core/evaluation/metric_utils.py","file_url":"https://github.com/researchmm/TTVSR/blob/HEAD/mmedit/core/evaluation/metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6140a018695b3869"}},{"code_sha256_prefix":"9e2a94b2a4a36877","entry":"gaussian","repo":"researchmm/TTVSR","repo_kind":"official","path":"mmedit/core/evaluation/metric_utils.py","file_url":"https://github.com/researchmm/TTVSR/blob/HEAD/mmedit/core/evaluation/metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e2a94b2a4a36877"}},{"code_sha256_prefix":"a593a9e8aabde3eb","entry":"pad_sequence","repo":"researchmm/TTVSR","repo_kind":"official","path":"mmedit/apis/restoration_video_inference.py","file_url":"https://github.com/researchmm/TTVSR/blob/HEAD/mmedit/apis/restoration_video_inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a593a9e8aabde3eb"}},{"code_sha256_prefix":"828da151bdfbb726","entry":"tensor2img","repo":"researchmm/TTVSR","repo_kind":"official","path":"mmedit/core/misc.py","file_url":"https://github.com/researchmm/TTVSR/blob/HEAD/mmedit/core/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"828da151bdfbb726"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}