{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/interaction2code-how-far-are-we-from","title":"Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping","arxiv_id":"2411.03292","date":"2024-11-05","proceeding":null,"authors":["Jingyu Xiao","Yuxuan Wan","Yintong Huo","Zixin Wang","Xinyi Xu","Wenxuan Wang","Zhiyao Xu","Yuhang Wang","Michael R. Lyu"],"abstract":"Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on the design-to-code task, i.e., generating UI code from UI mock-ups. However, existing benchmarks only contain static web pages for evaluation and ignore the dynamic interaction, limiting the practicality, usability and user engagement of the generated webpages. To bridge these gaps, we present the first systematic investigation of MLLMs in generating interactive webpages. Specifically, we formulate the Interaction-to-Code task and establish the Interaction2Code benchmark, encompassing 127 unique webpages and 374 distinct interactions across 15 webpage types and 31 interaction categories. Through comprehensive experiments utilizing state-of-the-art (SOTA) MLLMs, evaluated via both automatic metrics and human assessments, we identify four critical limitations of MLLM on Interaction-to-Code task: (1) inadequate generation of interaction compared with full page, (2) prone to ten types of failure, (3) poor performance on visually subtle interactions, and (4) insufficient undestanding on interaction when limited to single-modality visual descriptions. To address these limitations, we propose four enhancement strategies: Interactive Element Highlighting, Failureaware Prompting (FAP), Visual Saliency Enhancement, and Visual-Textual Descriptions Combination, all aiming at improving MLLMs' performance on the Interaction-toCode task. The Interaction2Code benchmark and code are available in https://github. com/WebPAI/Interaction2Code.","url_abs":"https://arxiv.org/abs/2411.03292v2","url_pdf":"https://arxiv.org/pdf/2411.03292v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"interaction2code-how-far-are-we-from","repo_url":"https://github.com/webpai/interaction2code","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.03292","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.03292"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/webpai/interaction2code","reach":{"status":"ok"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"8002d82f6f3348f4","entry":"clip_similarity","repo":"webpai/interaction2code","repo_kind":"official","path":"code/metric/calculate_metric.py","file_url":"https://github.com/webpai/interaction2code/blob/HEAD/code/metric/calculate_metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8002d82f6f3348f4"}},{"code_sha256_prefix":"6e35f046056a0d0c","entry":"convert_image_to_code","repo":"webpai/interaction2code","repo_kind":"official","path":"code/metric/metric_utils.py","file_url":"https://github.com/webpai/interaction2code/blob/HEAD/code/metric/metric_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e35f046056a0d0c"}},{"code_sha256_prefix":"89822a7f5aa26fcf","entry":"draw_rectangle","repo":"webpai/interaction2code","repo_kind":"official","path":"code/metric/metric_utils.py","file_url":"https://github.com/webpai/interaction2code/blob/HEAD/code/metric/metric_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89822a7f5aa26fcf"}},{"code_sha256_prefix":"a0f0e30c049d98f7","entry":"get_data","repo":"webpai/interaction2code","repo_kind":"official","path":"human_evaluation/Figure6_plot_compare.py","file_url":"https://github.com/webpai/interaction2code/blob/HEAD/human_evaluation/Figure6_plot_compare.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a0f0e30c049d98f7"}},{"code_sha256_prefix":"5ae32525b8c9e222","entry":"input_text","repo":"webpai/interaction2code","repo_kind":"official","path":"code/annotation/anno_interaction.py","file_url":"https://github.com/webpai/interaction2code/blob/HEAD/code/annotation/anno_interaction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ae32525b8c9e222"}},{"code_sha256_prefix":"07ce46f03ae3cad0","entry":"process_failure","repo":"webpai/interaction2code","repo_kind":"official","path":"human_evaluation/Figure6_plot_use_rate.py","file_url":"https://github.com/webpai/interaction2code/blob/HEAD/human_evaluation/Figure6_plot_use_rate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07ce46f03ae3cad0"}},{"code_sha256_prefix":"ce13f063f10284c1","entry":"read_file","repo":"webpai/interaction2code","repo_kind":"official","path":"code/metric/metric_utils.py","file_url":"https://github.com/webpai/interaction2code/blob/HEAD/code/metric/metric_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce13f063f10284c1"}},{"code_sha256_prefix":"46eb0083c8968174","entry":"ssim_similarity","repo":"webpai/interaction2code","repo_kind":"official","path":"code/metric/calculate_metric.py","file_url":"https://github.com/webpai/interaction2code/blob/HEAD/code/metric/calculate_metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46eb0083c8968174"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}