{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-language-models-for-automated-open","title":"Large Language Models for Automated Open-domain Scientific Hypotheses Discovery","arxiv_id":"2309.02726","date":"2023-09-06","proceeding":null,"authors":["Zonglin Yang","Xinya Du","Junxian Li","Jie Zheng","Soujanya Poria","Erik Cambria"],"abstract":"Hypothetical induction is recognized as the main reasoning type when scientists make observations about the world and try to propose hypotheses to explain those observations. Past research on hypothetical induction is under a constrained setting: (1) the observation annotations in the dataset are carefully manually handpicked sentences (resulting in a close-domain setting); and (2) the ground truth hypotheses are mostly commonsense knowledge, making the task less challenging. In this work, we tackle these problems by proposing the first dataset for social science academic hypotheses discovery, with the final goal to create systems that automatically generate valid, novel, and helpful scientific hypotheses, given only a pile of raw web corpus. Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity. A multi-module framework is developed for the task, including three different feedback mechanisms to boost performance, which exhibits superior performance in terms of both GPT-4 based and expert-based evaluation. To the best of our knowledge, this is the first work showing that LLMs are able to generate novel (''not existing in literature'') and valid (''reflecting reality'') scientific hypotheses.","url_abs":"https://arxiv.org/abs/2309.02726v3","url_pdf":"https://arxiv.org/pdf/2309.02726v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"large-language-models-for-automated-open","repo_url":"https://github.com/zongliny/moose","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"valid"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.02726","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.02726"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zongliny/moose","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":5},"by_repo_kind":{"official":{"samples":11,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"d339db3d3535b42d","entry":"consistency","repo":"zongliny/moose","repo_kind":"official","path":"consistency_between_expert_gpt4.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/consistency_between_expert_gpt4.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d339db3d3535b42d"}},{"code_sha256_prefix":"2973a534de078a80","entry":"find_score","repo":"zongliny/moose","repo_kind":"official","path":"compare_score.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/compare_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2973a534de078a80"}},{"code_sha256_prefix":"422538e97c5a005c","entry":"pick_score","repo":"zongliny/moose","repo_kind":"official","path":"evaluate_utils.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/evaluate_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"422538e97c5a005c"}},{"code_sha256_prefix":"354d54b9bc92699d","entry":"read_file_find_score_concat_score","repo":"zongliny/moose","repo_kind":"official","path":"compare_score.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/compare_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"354d54b9bc92699d"}},{"code_sha256_prefix":"5f5c9cf8262477b9","entry":"read_score_file","repo":"zongliny/moose","repo_kind":"official","path":"compare_score.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/compare_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f5c9cf8262477b9"}},{"code_sha256_prefix":"8f55a8eb680be1a0","entry":"unify_gpt4_scores","repo":"zongliny/moose","repo_kind":"official","path":"consistency_between_expert_gpt4.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/consistency_between_expert_gpt4.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f55a8eb680be1a0"}},{"code_sha256_prefix":"a537a7078b759aeb","entry":"final_result","repo":"zongliny/moose","repo_kind":"official","path":"read_expert_eval.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/read_expert_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a537a7078b759aeb"}},{"code_sha256_prefix":"6c869e40b7b5c531","entry":"load_annotated_data","repo":"zongliny/moose","repo_kind":"official","path":"read_expert_eval.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/read_expert_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c869e40b7b5c531"}},{"code_sha256_prefix":"f9f3334343bb7080","entry":"load_gpt4_scores","repo":"zongliny/moose","repo_kind":"official","path":"consistency_between_expert_gpt4.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/consistency_between_expert_gpt4.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9f3334343bb7080"}},{"code_sha256_prefix":"b222dce236f33733","entry":"load_ground_truth_hypotheses","repo":"zongliny/moose","repo_kind":"official","path":"evaluate_utils.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/evaluate_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b222dce236f33733"}},{"code_sha256_prefix":"e270057b4b2f7076","entry":"random_pick","repo":"zongliny/moose","repo_kind":"official","path":"pick_hyp_for_expert_eval.py","file_url":"https://github.com/zongliny/moose/blob/HEAD/pick_hyp_for_expert_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e270057b4b2f7076"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}