{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/massw-a-new-dataset-and-benchmark-tasks-for","title":"MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows","arxiv_id":"2406.06357","date":"2024-06-10","proceeding":null,"authors":["Xingjian Zhang","Yutong Xie","Jin Huang","Jinge Ma","Zhaoying Pan","Qijia Liu","Ziyang Xiong","Tolga Ergen","Dongsub Shim","Honglak Lee","Qiaozhu Mei"],"abstract":"Scientific innovation relies on detailed workflows, which include critical steps such as analyzing literature, generating ideas, validating these ideas, interpreting results, and inspiring follow-up research. However, scientific publications that document these workflows are extensive and unstructured. This makes it difficult for both human researchers and AI systems to effectively navigate and explore the space of scientific innovation. To address this issue, we introduce MASSW, a comprehensive text dataset on Multi-Aspect Summarization of Scientific Workflows. MASSW includes more than 152,000 peer-reviewed publications from 17 leading computer science conferences spanning the past 50 years. Using Large Language Models (LLMs), we automatically extract five core aspects from these publications -- context, key idea, method, outcome, and projected impact -- which correspond to five key steps in the research workflow. These structured summaries facilitate a variety of downstream tasks and analyses. The quality of the LLM-extracted summaries is validated by comparing them with human annotations. We demonstrate the utility of MASSW through multiple novel machine-learning tasks that can be benchmarked using this new dataset, which make various types of predictions and recommendations along the scientific workflow. MASSW holds significant potential for researchers to create and benchmark new AI methods for optimizing scientific workflows and fostering scientific innovation in the field. Our dataset is openly available at \\url{https://github.com/xingjian-zhang/massw}.","url_abs":"https://arxiv.org/abs/2406.06357v1","url_pdf":"https://arxiv.org/pdf/2406.06357v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"massw-a-new-dataset-and-benchmark-tasks-for","repo_url":"https://github.com/xingjian-zhang/massw","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"CC0-1.0"}}],"tasks":[{"task_slug":"navigate","task_name":"Navigate"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.06357","atlas_url":"https://app.syntology.ai/?focus=2406.06357","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.06357"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xingjian-zhang/massw","reach":{"status":"ok","spdx":"CC0-1.0"}}],"summary":{"ran":10,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ad034ce77d764f7d","entry":"add_messages_and_run","repo":"xingjian-zhang/massw","repo_kind":"official","path":"massw/models/gpt_azure.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/massw/models/gpt_azure.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad034ce77d764f7d"}},{"code_sha256_prefix":"510528f4db1006bc","entry":"best_demos","repo":"xingjian-zhang/massw","repo_kind":"official","path":"factscore/atomic_facts.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/factscore/atomic_facts.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"510528f4db1006bc"}},{"code_sha256_prefix":"65b281d7162641cc","entry":"flatten_metrics","repo":"xingjian-zhang/massw","repo_kind":"official","path":"massw/metrics.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/massw/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65b281d7162641cc"}},{"code_sha256_prefix":"387f336b1c4b3c25","entry":"handle_mixtral_model","repo":"xingjian-zhang/massw","repo_kind":"official","path":"massw/models/mixtral_azure.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/massw/models/mixtral_azure.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"387f336b1c4b3c25"}},{"code_sha256_prefix":"8eb846a8e76a79b8","entry":"is_invalid_paragraph_ppl","repo":"xingjian-zhang/massw","repo_kind":"official","path":"factscore/abstain_detection.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/factscore/abstain_detection.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8eb846a8e76a79b8"}},{"code_sha256_prefix":"672bd38d830be48d","entry":"is_invalid_ppl","repo":"xingjian-zhang/massw","repo_kind":"official","path":"factscore/abstain_detection.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/factscore/abstain_detection.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"672bd38d830be48d"}},{"code_sha256_prefix":"6a96435eba311b08","entry":"normalize_answer","repo":"xingjian-zhang/massw","repo_kind":"official","path":"factscore/atomic_facts.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/factscore/atomic_facts.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a96435eba311b08"}},{"code_sha256_prefix":"8732e3cee65072f2","entry":"prompts_to_raw_output","repo":"xingjian-zhang/massw","repo_kind":"official","path":"massw/models/mixtral_azure.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/massw/models/mixtral_azure.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8732e3cee65072f2"}},{"code_sha256_prefix":"8cdcb69190dee2a5","entry":"raw_output_to_dict","repo":"xingjian-zhang/massw","repo_kind":"official","path":"massw/models/mixtral_azure.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/massw/models/mixtral_azure.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8cdcb69190dee2a5"}},{"code_sha256_prefix":"3635046105ae0fce","entry":"remove_citation","repo":"xingjian-zhang/massw","repo_kind":"official","path":"factscore/abstain_detection.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/factscore/abstain_detection.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3635046105ae0fce"}},{"code_sha256_prefix":"13867f872a481bd3","entry":"text_to_sentences","repo":"xingjian-zhang/massw","repo_kind":"official","path":"factscore/atomic_facts.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/factscore/atomic_facts.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13867f872a481bd3"}},{"code_sha256_prefix":"add7cc2c75fac6a5","entry":"prompts_to_raw_output","repo":"xingjian-zhang/massw","repo_kind":"official","path":"massw/models/gpt_azure.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/massw/models/gpt_azure.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"add7cc2c75fac6a5"}},{"code_sha256_prefix":"c804619a1c095b59","entry":"raw_output_to_dict","repo":"xingjian-zhang/massw","repo_kind":"official","path":"massw/models/gpt_azure.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/massw/models/gpt_azure.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c804619a1c095b59"}},{"code_sha256_prefix":"02c83d89394be1c7","entry":"recover_instruct_llama","repo":"xingjian-zhang/massw","repo_kind":"official","path":"factscore/download_data.py","file_url":"https://github.com/xingjian-zhang/massw/blob/HEAD/factscore/download_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"02c83d89394be1c7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}