{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/subjective-aligned-dateset-and-metric-for","title":"Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment","arxiv_id":"2403.11956","date":"2024-03-18","proceeding":null,"authors":["Tengchuan Kou","Xiaohong Liu","ZiCheng Zhang","Chunyi Li","HaoNing Wu","Xiongkuo Min","Guangtao Zhai","Ning Liu"],"abstract":"With the rapid development of generative models, Artificial Intelligence-Generated Contents (AIGC) have exponentially increased in daily lives. Among them, Text-to-Video (T2V) generation has received widespread attention. Though many T2V models have been released for generating high perceptual quality videos, there is still lack of a method to evaluate the quality of these videos quantitatively. To solve this issue, we establish the largest-scale Text-to-Video Quality Assessment DataBase (T2VQA-DB) to date. The dataset is composed of 10,000 videos generated by 9 different T2V models. We also conduct a subjective study to obtain each video's corresponding mean opinion score. Based on T2VQA-DB, we propose a novel transformer-based model for subjective-aligned Text-to-Video Quality Assessment (T2VQA). The model extracts features from text-video alignment and video fidelity perspectives, then it leverages the ability of a large language model to give the prediction score. Experimental results show that T2VQA outperforms existing T2V metrics and SOTA video quality assessment models. Quantitative analysis indicates that T2VQA is capable of giving subjective-align predictions, validating its effectiveness. The dataset and code will be released at https://github.com/QMME/T2VQA.","url_abs":"https://arxiv.org/abs/2403.11956v5","url_pdf":"https://arxiv.org/pdf/2403.11956v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"subjective-aligned-dateset-and-metric-for","repo_url":"https://github.com/qmme/t2vqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"video-alignment","task_name":"Video Alignment"},{"task_slug":"video-quality-assessment","task_name":"Video Quality Assessment"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.11956","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.11956"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qmme/t2vqa","reach":{"status":"ok"}}],"summary":{"ran_violates":1,"ran_honours":1,"ran":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"qmme/t2vqa","repo_kind":"official","path":"model/model.py","file_url":"https://github.com/qmme/t2vqa/blob/HEAD/model/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"e4f140f85865ac5e","entry":"fragment_infos","repo":"qmme/t2vqa","repo_kind":"official","path":"model/swin.py","file_url":"https://github.com/qmme/t2vqa/blob/HEAD/model/swin.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4f140f85865ac5e"}},{"code_sha256_prefix":"3ec6d5c28e50e381","entry":"get_adaptive_window_size","repo":"qmme/t2vqa","repo_kind":"official","path":"model/swin.py","file_url":"https://github.com/qmme/t2vqa/blob/HEAD/model/swin.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ec6d5c28e50e381"}},{"code_sha256_prefix":"6a72c202cbcb5ab5","entry":"global_position_index","repo":"qmme/t2vqa","repo_kind":"official","path":"model/swin.py","file_url":"https://github.com/qmme/t2vqa/blob/HEAD/model/swin.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a72c202cbcb5ab5"}},{"code_sha256_prefix":"c6ec173f19f5c34d","entry":"interpolate_pos_embed","repo":"qmme/t2vqa","repo_kind":"official","path":"model/vit.py","file_url":"https://github.com/qmme/t2vqa/blob/HEAD/model/vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6ec173f19f5c34d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}