{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scalable-video-to-dataset-generation-for","title":"Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents","arxiv_id":"2505.12632","date":"2025-05-19","proceeding":"CVPR 2025 1","authors":["Yunseok Jang","Yeda Song","Sungryull Sohn","Lajanugen Logeswaran","Tiange Luo","Dong-Ki Kim","Kyunghoon Bae","Honglak Lee"],"abstract":"Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have sparked significant interest in developing GUI visual agents. We introduce MONDAY (Mobile OS Navigation Task Dataset for Agents from YouTube), a large-scale dataset of 313K annotated frames from 20K instructional videos capturing diverse real-world mobile OS navigation across multiple platforms. Models that include MONDAY in their pre-training phases demonstrate robust cross-platform generalization capabilities, consistently outperforming models trained on existing single OS datasets while achieving an average performance gain of 18.11%p on an unseen mobile OS platform. To enable continuous dataset expansion as mobile platforms evolve, we present an automated framework that leverages publicly available video content to create comprehensive task datasets without manual annotation. Our framework comprises robust OCR-based scene detection (95.04% F1score), near-perfect UI element detection (99.87% hit ratio), and novel multi-step action identification to extract reliable action sequences across diverse interface configurations. We contribute both the MONDAY dataset and our automated collection framework to facilitate future research in mobile OS navigation.","url_abs":"https://arxiv.org/abs/2505.12632v1","url_pdf":"https://arxiv.org/pdf/2505.12632v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"dataset-generation","task_name":"Dataset Generation"},{"task_slug":"optical-character-recognition","task_name":"Optical Character Recognition (OCR)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.12632","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.12632"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/runamu/monday","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":8},"by_repo_kind":{"found_in_text":{"samples":9,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"41f6e1b0feaded2f","entry":"is_tap_action","repo":"runamu/monday","repo_kind":"found_in_text","path":"agent_tasks/action_matching.py","file_url":"https://github.com/runamu/monday/blob/HEAD/agent_tasks/action_matching.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41f6e1b0feaded2f"}},{"code_sha256_prefix":"42dd46cf39372b94","entry":"action2step_common","repo":"runamu/monday","repo_kind":"found_in_text","path":"agent_tasks/eval_seeclick.py","file_url":"https://github.com/runamu/monday/blob/HEAD/agent_tasks/eval_seeclick.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42dd46cf39372b94"}},{"code_sha256_prefix":"ce5baa687c42285a","entry":"action_2_format","repo":"runamu/monday","repo_kind":"found_in_text","path":"agent_tasks/action_matching.py","file_url":"https://github.com/runamu/monday/blob/HEAD/agent_tasks/action_matching.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce5baa687c42285a"}},{"code_sha256_prefix":"9904a95483802677","entry":"format_seconds","repo":"runamu/monday","repo_kind":"found_in_text","path":"data_processing/utils.py","file_url":"https://github.com/runamu/monday/blob/HEAD/data_processing/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9904a95483802677"}},{"code_sha256_prefix":"3259fb8a9fc8234b","entry":"get_pure_size","repo":"runamu/monday","repo_kind":"found_in_text","path":"data_processing/utils.py","file_url":"https://github.com/runamu/monday/blob/HEAD/data_processing/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3259fb8a9fc8234b"}},{"code_sha256_prefix":"4d074d11cb63be1d","entry":"get_seeclick_model","repo":"runamu/monday","repo_kind":"found_in_text","path":"agent_tasks/eval_seeclick.py","file_url":"https://github.com/runamu/monday/blob/HEAD/agent_tasks/eval_seeclick.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4d074d11cb63be1d"}},{"code_sha256_prefix":"52ceba3e2a19a07b","entry":"get_seeclick_response","repo":"runamu/monday","repo_kind":"found_in_text","path":"agent_tasks/eval_seeclick.py","file_url":"https://github.com/runamu/monday/blob/HEAD/agent_tasks/eval_seeclick.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52ceba3e2a19a07b"}},{"code_sha256_prefix":"c9a868b25b751c41","entry":"process_dataset","repo":"runamu/monday","repo_kind":"found_in_text","path":"data_processing/extract_scenes.py","file_url":"https://github.com/runamu/monday/blob/HEAD/data_processing/extract_scenes.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9a868b25b751c41"}},{"code_sha256_prefix":"f0ee53e5b2feb7e6","entry":"process_row","repo":"runamu/monday","repo_kind":"found_in_text","path":"data_processing/extract_scenes.py","file_url":"https://github.com/runamu/monday/blob/HEAD/data_processing/extract_scenes.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0ee53e5b2feb7e6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}