{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/viotgpt-learning-to-schedule-vision-tools","title":"VIoTGPT: Learning to Schedule Vision Tools in LLMs towards Intelligent Video Internet of Things","arxiv_id":"2312.00401","date":"2023-12-01","proceeding":null,"authors":["Yaoyao Zhong","Mengshi Qi","Rui Wang","Yuhan Qiu","Yang Zhang","Huadong Ma"],"abstract":"Video Internet of Things (VIoT) has shown full potential in collecting an unprecedented volume of video data. How to schedule the domain-specific perceiving models and analyze the collected videos uniformly, efficiently, and especially intelligently to accomplish complicated tasks is challenging. To address the challenge, we build VIoTGPT, the framework based on LLMs to correctly interact with humans, query knowledge videos, and invoke vision models to analyze multimedia data collaboratively. To support VIoTGPT and related future works, we meticulously crafted the VIoT-Tool dataset, including the training dataset and the benchmark involving 11 representative vision models across three categories based on semi-automatic annotations. To guide LLM to act as the intelligent agent towards intelligent VIoT, we resort to the ReAct instruction tuning method based on VIoT-Tool to learn the tool capability. Quantitative and qualitative experiments and analyses demonstrate the effectiveness of VIoTGPT. We believe VIoTGPT contributes to improving human-centered experiences in VIoT applications. The project website is https://github.com/zhongyy/VIoTGPT.","url_abs":"https://arxiv.org/abs/2312.00401v2","url_pdf":"https://arxiv.org/pdf/2312.00401v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"viotgpt-learning-to-schedule-vision-tools","repo_url":"https://github.com/zhongyy/viotgpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.00401","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.00401"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhongyy/viotgpt","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_violates":1,"unverified":7},"by_repo_kind":{"official":{"samples":11,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"709a780a94ca2670","entry":"get_compressed_list","repo":"zhongyy/viotgpt","repo_kind":"official","path":"train/compression.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/train/compression.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"709a780a94ca2670"}},{"code_sha256_prefix":"e8bafcb096991087","entry":"prompts","repo":"zhongyy/viotgpt","repo_kind":"official","path":"VIoTGPT_Vision_nodemo.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/VIoTGPT_Vision_nodemo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8bafcb096991087"}},{"code_sha256_prefix":"0bf414a2b44babe6","entry":"raise_warning_for_incompatible_cpu_offloading_configuration","repo":"zhongyy/viotgpt","repo_kind":"official","path":"train/model_adapter.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/train/model_adapter.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bf414a2b44babe6"}},{"code_sha256_prefix":"69d44b0bc956fb78","entry":"visualize","repo":"zhongyy/viotgpt","repo_kind":"official","path":"VIoTGPT_Vision_nodemo.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/VIoTGPT_Vision_nodemo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69d44b0bc956fb78"}},{"code_sha256_prefix":"3408dd47e939d760","entry":"apply_rotary_pos_emb","repo":"zhongyy/viotgpt","repo_kind":"official","path":"train/llama2_flash_attn_monkey_patch.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/train/llama2_flash_attn_monkey_patch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3408dd47e939d760"}},{"code_sha256_prefix":"9eccbab8ec3b8244","entry":"build_logger","repo":"zhongyy/viotgpt","repo_kind":"official","path":"train/utils.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/train/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9eccbab8ec3b8244"}},{"code_sha256_prefix":"386536152177a379","entry":"compress","repo":"zhongyy/viotgpt","repo_kind":"official","path":"train/compression.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/train/compression.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"386536152177a379"}},{"code_sha256_prefix":"9cea3d56fecb304b","entry":"get_conv_template","repo":"zhongyy/viotgpt","repo_kind":"official","path":"train/tool_conversation.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/train/tool_conversation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9cea3d56fecb304b"}},{"code_sha256_prefix":"2f3cb5d0dcc18502","entry":"get_gpu_memory","repo":"zhongyy/viotgpt","repo_kind":"official","path":"train/utils.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/train/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f3cb5d0dcc18502"}},{"code_sha256_prefix":"5f9d4d17fb70af2b","entry":"load_compress_model","repo":"zhongyy/viotgpt","repo_kind":"official","path":"train/compression.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/train/compression.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f9d4d17fb70af2b"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"zhongyy/viotgpt","repo_kind":"official","path":"train/utils.py","file_url":"https://github.com/zhongyy/viotgpt/blob/HEAD/train/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}