{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/howto100m-learning-a-text-video-embedding-by","title":"HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips","arxiv_id":"1906.03327","date":"2019-06-07","proceeding":"ICCV 2019 10","authors":["Antoine Miech","Dimitri Zhukov","Jean-Baptiste Alayrac","Makarand Tapaswi","Ivan Laptev","Josef Sivic"],"abstract":"Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models will be publicly available at: www.di.ens.fr/willow/research/howto100m/.","url_abs":"https://arxiv.org/abs/1906.03327v2","url_pdf":"https://arxiv.org/pdf/1906.03327v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"howto100m-learning-a-text-video-embedding-by","repo_url":"https://github.com/antoine77340/MIL-NCE_HowTo100M","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"howto100m-learning-a-text-video-embedding-by","repo_url":"https://github.com/antoine77340/S3D_HowTo100M","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"howto100m-learning-a-text-video-embedding-by","repo_url":"https://github.com/antoine77340/milnce_howto100m","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"howto100m-learning-a-text-video-embedding-by","repo_url":"https://github.com/roudimit/AVLnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"action-localization","task_name":"Action Localization"},{"task_slug":"long-video-retrieval-background-removed","task_name":"Long Video Retrieval (Background Removed)"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-to-video-retrieval","task_name":"Text to Video Retrieval"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[],"datasets_introduced":[{"slug":"howto100m","name":"HowTo100M","full_name":"HowTo100M"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/long-video-retrieval-background-removed-on","task":"Long Video Retrieval (Background Removed)","dataset":"YouCook2","model":"Text-Video Embedding","rank_in_archive_order":5,"of":6,"metrics":{"Cap. Avg. R@1":"46.6","Cap. Avg. R@10":"83.7","Cap. Avg. R@5":"74.3"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-crosstask","task":"Temporal Action Localization","dataset":"CrossTask","model":"Text-Video Embedding","rank_in_archive_order":4,"of":7,"metrics":{"Recall":"33.6"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"Text-Video Embedding","rank_in_archive_order":36,"of":38,"metrics":{"text-to-video Median Rank":"40","text-to-video R@1":"7.2","text-to-video R@10":"27.9","text-to-video R@5":"19.6"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt","task":"Video Retrieval","dataset":"MSR-VTT","model":"Text-Video Embedding","rank_in_archive_order":34,"of":40,"metrics":{"text-to-video Median Rank":"9","text-to-video R@1":"14.9","text-to-video R@10":"52.8","video-to-text R@5":"40.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"HT-Pretrained","rank_in_archive_order":57,"of":63,"metrics":{"text-to-video Median Rank":"9","text-to-video R@1":"14.9","text-to-video R@10":"52.8","text-to-video R@5":"40.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"HT","rank_in_archive_order":58,"of":63,"metrics":{"text-to-video Median Rank":"12","text-to-video R@1":"12.1","text-to-video R@10":"48.0","text-to-video R@5":"35.0"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-youcook2","task":"Video Retrieval","dataset":"YouCook2","model":"Text-Video Embedding","rank_in_archive_order":11,"of":16,"metrics":{"text-to-video Median Rank":"24","text-to-video R@1":"8.2","text-to-video R@10":"35.3","text-to-video R@5":"24.5"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1906.03327","atlas_url":"https://app.syntology.ai/?focus=1906.03327","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.03327"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/antoine77340/S3D_HowTo100M","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/antoine77340/milnce_howto100m","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/antoine77340/MIL-NCE_HowTo100M","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/roudimit/AVLnet","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"4d241892c1ca731a","entry":"get_last_checkpoint","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"4d241892c1ca731a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}