{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/do-multimodal-foundation-models-understand","title":"WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Tasks","arxiv_id":"2406.13264","date":"2024-06-19","proceeding":null,"authors":["Michael Wornow","Avanika Narayan","Ben Viggiano","Ishan S. Khare","Tathagat Verma","Tibor Thompson","Miguel Angel Fuentes Hernandez","Sudharsan Sundar","Chloe Trujillo","Krrish Chawla","Rongfei Lu","Justin Shen","Divya Nagaraj","Joshua Martinez","Vardhan Agrawal","Althea Hudson","Nigam H. Shah","Christopher Re"],"abstract":"Existing ML benchmarks lack the depth and diversity of annotations needed for evaluating models on business process management (BPM) tasks. BPM is the practice of documenting, measuring, improving, and automating enterprise workflows. However, research has focused almost exclusively on one task - full end-to-end automation using agents based on multimodal foundation models (FMs) like GPT-4. This focus on automation ignores the reality of how most BPM tools are applied today - simply documenting the relevant workflow takes 60% of the time of the typical process optimization project. To address this gap we present WONDERBREAD, the first benchmark for evaluating multimodal FMs on BPM tasks beyond automation. Our contributions are: (1) a dataset containing 2928 documented workflow demonstrations; (2) 6 novel BPM tasks sourced from real-world applications ranging from workflow documentation to knowledge transfer to process improvement; and (3) an automated evaluation harness. Our benchmark shows that while state-of-the-art FMs can automatically generate documentation (e.g. recalling 88% of the steps taken in a video demonstration of a workflow), they struggle to re-apply that knowledge towards finer-grained validation of workflow completion (F1 < 0.3). We hope WONDERBREAD encourages the development of more \"human-centered\" AI tooling for enterprise applications and furthers the exploration of multimodal FMs for the broader universe of BPM tasks. We publish our dataset and experiments here: https://github.com/HazyResearch/wonderbread","url_abs":"https://arxiv.org/abs/2406.13264v2","url_pdf":"https://arxiv.org/pdf/2406.13264v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"do-multimodal-foundation-models-understand","repo_url":"https://github.com/hazyresearch/wonderbread","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"management","task_name":"Management"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.13264","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.13264"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/wonderbread","reach":{"status":"ok"}}],"summary":{"ran":11,"unverified":1},"by_repo_kind":{"official":{"samples":12,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"6775327d20e8cb7e","entry":"extract_folder_id","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/collect/download.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/collect/download.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6775327d20e8cb7e"}},{"code_sha256_prefix":"0c89ee73a830bd09","entry":"gen_ablation_name","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/benchmark/eval/run_demo_segmentation.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/benchmark/eval/run_demo_segmentation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c89ee73a830bd09"}},{"code_sha256_prefix":"0b1c198518518777","entry":"gen_ablation_name","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/benchmark/eval/run_sop_generation.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/benchmark/eval/run_sop_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b1c198518518777"}},{"code_sha256_prefix":"19580b844f58d75d","entry":"get_files_in_folder","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/helpers.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"19580b844f58d75d"}},{"code_sha256_prefix":"cdef0dcde3c15ae2","entry":"get_folder_id_from_gdrive_link","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/collect/get_drive_folder_names.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/collect/get_drive_folder_names.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cdef0dcde3c15ae2"}},{"code_sha256_prefix":"38929d9e6f23feba","entry":"get_folder_name","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/collect/download.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/collect/download.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38929d9e6f23feba"}},{"code_sha256_prefix":"0cc1d82b07ef63eb","entry":"get_folders_in_folder","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/helpers.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0cc1d82b07ef63eb"}},{"code_sha256_prefix":"a310a7faed0cf706","entry":"get_optimal_label_assignment","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/benchmark/eval/run_demo_segmentation.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/benchmark/eval/run_demo_segmentation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a310a7faed0cf706"}},{"code_sha256_prefix":"a8e4397f01df9f73","entry":"is_string_in_integer_range","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/collect/record.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/collect/record.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8e4397f01df9f73"}},{"code_sha256_prefix":"a9a200c442872344","entry":"process_row","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/collect/download.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/collect/download.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9a200c442872344"}},{"code_sha256_prefix":"f84097ca1e5bf196","entry":"rubric_evaluation_prompt","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/benchmark/eval/eval_rubric.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/benchmark/eval/eval_rubric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f84097ca1e5bf196"}},{"code_sha256_prefix":"fe4f7602b1a9b1e5","entry":"create_merged_trace","repo":"hazyresearch/wonderbread","repo_kind":"official","path":"wonderbread/helpers.py","file_url":"https://github.com/hazyresearch/wonderbread/blob/HEAD/wonderbread/helpers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe4f7602b1a9b1e5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}