{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-the-evaluation-of-video-summaries","title":"Rethinking the Evaluation of Video Summaries","arxiv_id":"1903.11328","date":"2019-03-27","proceeding":"CVPR 2019 6","authors":["Mayu Otani","Yuta Nakashima","Esa Rahtu","Janne Heikkilä"],"abstract":"Video summarization is a technique to create a short skim of the original\nvideo while preserving the main stories/content. There exists a substantial\ninterest in automatizing this process due to the rapid growth of the available\nmaterial. The recent progress has been facilitated by public benchmark\ndatasets, which enable easy and fair comparison of methods. Currently the\nestablished evaluation protocol is to compare the generated summary with\nrespect to a set of reference summaries provided by the dataset. In this paper,\nwe will provide in-depth assessment of this pipeline using two popular\nbenchmark datasets. Surprisingly, we observe that randomly generated summaries\nachieve comparable or better performance to the state-of-the-art. In some\ncases, the random summaries outperform even the human generated summaries in\nleave-one-out experiments. Moreover, it turns out that the video segmentation,\nwhich is often considered as a fixed pre-processing method, has the most\nsignificant impact on the performance measure. Based on our observations, we\npropose alternative approaches for assessing the importance scores as well as\nan intuitive visualization of correlation between the estimated scoring and\nhuman annotations.","url_abs":"http://arxiv.org/abs/1903.11328v2","url_pdf":"http://arxiv.org/pdf/1903.11328v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-the-evaluation-of-video-summaries","repo_url":"https://github.com/emsalinha/videosum-eval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"rethinking-the-evaluation-of-video-summaries","repo_url":"https://github.com/mayu-ot/rethinking-evs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"video-segmentation","task_name":"Video Segmentation"},{"task_slug":"video-semantic-segmentation","task_name":"Video Semantic Segmentation"},{"task_slug":"video-summarization","task_name":"Video Summarization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1903.11328","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1903.11328"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/emsalinha/videosum-eval","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mayu-ot/rethinking-evs","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":1},"by_repo_kind":{"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e9cdc6271927cde1","entry":"convert_time","repo":"emsalinha/videosum-eval","repo_kind":"listed","path":"play_summary/create_summaries.py","file_url":"https://github.com/emsalinha/videosum-eval/blob/HEAD/play_summary/create_summaries.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9cdc6271927cde1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}