{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/myriad-large-multimodal-model-by-applying","title":"Myriad: Large Multimodal Model by Applying Vision Experts for Industrial Anomaly Detection","arxiv_id":"2310.19070","date":"2023-10-29","proceeding":null,"authors":["Yuanze Li","Haolin Wang","Shihao Yuan","Ming Liu","Debin Zhao","Yiwen Guo","Chen Xu","Guangming Shi","WangMeng Zuo"],"abstract":"Due to the training configuration, traditional industrial anomaly detection (IAD) methods have to train a specific model for each deployment scenario, which is insufficient to meet the requirements of modern design and manufacturing. On the contrary, large multimodal models~(LMMs) have shown eminent generalization ability on various vision tasks, and their perception and comprehension capabilities imply the potential of applying LMMs on IAD tasks. However, we observe that even though the LMMs have abundant knowledge about industrial anomaly detection in the textual domain, the LMMs are unable to leverage the knowledge due to the modality gap between textual and visual domains. To stimulate the relevant knowledge in LMMs and adapt the LMMs towards anomaly detection tasks, we introduce existing IAD methods as vision experts and present a novel large multimodal model applying vision experts for industrial anomaly detection~(abbreviated to {Myriad}). Specifically, we utilize the anomaly map generated by the vision experts as guidance for LMMs, such that the vision model is guided to pay more attention to anomalous regions. Then, the visual features are modulated via an adapter to fit the anomaly detection tasks, which are fed into the language model together with the vision expert guidance and human instructions to generate the final outputs. Extensive experiments are applied on MVTec-AD, VisA, and PCB Bank benchmarks demonstrate that our proposed method not only performs favorably against state-of-the-art methods, but also inherits the flexibility and instruction-following ability of LMMs in the field of IAD. Source code and pre-trained models are publicly available at \\url{https://github.com/tzjtatata/Myriad}.","url_abs":"https://arxiv.org/abs/2310.19070v3","url_pdf":"https://arxiv.org/pdf/2310.19070v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"myriad-large-multimodal-model-by-applying","repo_url":"https://github.com/tzjtatata/myriad","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"anomaly-detection","task_name":"Anomaly Detection"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.19070","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.19070"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tzjtatata/myriad","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_violates":2,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"cb41c9f77c7bd156","entry":"anomaly_map_handler","repo":"tzjtatata/myriad","repo_kind":"official","path":"evaluation_aqa_dataset.py","file_url":"https://github.com/tzjtatata/myriad/blob/HEAD/evaluation_aqa_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb41c9f77c7bd156"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"tzjtatata/myriad","repo_kind":"official","path":"minigpt4/models/blip2.py","file_url":"https://github.com/tzjtatata/myriad/blob/HEAD/minigpt4/models/blip2.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"b902e006ef57cdcb","entry":"find_first_file_in_directory","repo":"tzjtatata/myriad","repo_kind":"official","path":"minigpt4/models/adrefexpert_v2.py","file_url":"https://github.com/tzjtatata/myriad/blob/HEAD/minigpt4/models/adrefexpert_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b902e006ef57cdcb"}},{"code_sha256_prefix":"6ba8cee9f5daea41","entry":"pair","repo":"tzjtatata/myriad","repo_kind":"official","path":"minigpt4/models/fuser.py","file_url":"https://github.com/tzjtatata/myriad/blob/HEAD/minigpt4/models/fuser.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ba8cee9f5daea41"}},{"code_sha256_prefix":"cfe18b9cc8afbfe9","entry":"posemb_sincos_2d","repo":"tzjtatata/myriad","repo_kind":"official","path":"minigpt4/models/fuser.py","file_url":"https://github.com/tzjtatata/myriad/blob/HEAD/minigpt4/models/fuser.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfe18b9cc8afbfe9"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"tzjtatata/myriad","repo_kind":"official","path":"minigpt4/models/base_model.py","file_url":"https://github.com/tzjtatata/myriad/blob/HEAD/minigpt4/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"tzjtatata/myriad","repo_kind":"official","path":"minigpt4/models/base_model.py","file_url":"https://github.com/tzjtatata/myriad/blob/HEAD/minigpt4/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"b7db790f54abfb17","entry":"encode_text_with_prompt_ensemble","repo":"tzjtatata/myriad","repo_kind":"official","path":"minigpt4/models/adexpert.py","file_url":"https://github.com/tzjtatata/myriad/blob/HEAD/minigpt4/models/adexpert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7db790f54abfb17"}},{"code_sha256_prefix":"649c22213d7b3f54","entry":"get_context_emb","repo":"tzjtatata/myriad","repo_kind":"official","path":"evaluation_aqa_dataset.py","file_url":"https://github.com/tzjtatata/myriad/blob/HEAD/evaluation_aqa_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"649c22213d7b3f54"}},{"code_sha256_prefix":"a07ad5ab4092f40e","entry":"get_conv_act","repo":"tzjtatata/myriad","repo_kind":"official","path":"minigpt4/models/fuser.py","file_url":"https://github.com/tzjtatata/myriad/blob/HEAD/minigpt4/models/fuser.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a07ad5ab4092f40e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}