{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-large-language-models-unlock-novel","title":"Can Large Language Models Unlock Novel Scientific Research Ideas?","arxiv_id":"2409.06185","date":"2024-09-10","proceeding":null,"authors":["Sandeep Kumar","Tirthankar Ghosal","Vinayak Goyal","Asif Ekbal"],"abstract":"\"An idea is nothing more nor less than a new combination of old elements\" (Young, J.W.). The widespread adoption of Large Language Models (LLMs) and publicly available ChatGPT have marked a significant turning point in the integration of Artificial Intelligence (AI) into people's everyday lives. This study explores the capability of LLMs in generating novel research ideas based on information from research papers. We conduct a thorough examination of 4 LLMs in five domains (e.g., Chemistry, Computer, Economics, Medical, and Physics). We found that the future research ideas generated by Claude-2 and GPT-4 are more aligned with the author's perspective than GPT-3.5 and Gemini. We also found that Claude-2 generates more diverse future research ideas than GPT-4, GPT-3.5, and Gemini 1.0. We further performed a human evaluation of the novelty, relevancy, and feasibility of the generated future research ideas. This investigation offers insights into the evolving role of LLMs in idea generation, highlighting both its capability and limitations. Our work contributes to the ongoing efforts in evaluating and utilizing language models for generating future research ideas. We make our datasets and codes publicly available.","url_abs":"https://arxiv.org/abs/2409.06185v1","url_pdf":"https://arxiv.org/pdf/2409.06185v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-large-language-models-unlock-novel","repo_url":"https://github.com/sandeep82945/future-idea-generation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2409.06185","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.06185"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sandeep82945/future-idea-generation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":9},"by_repo_kind":{"official":{"samples":11,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c426f422d4b0971","entry":"remove_brackets","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/preprocessing.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c426f422d4b0971"}},{"code_sha256_prefix":"8fc79d192167311d","entry":"text_to_list","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/GPT_as_reviewer.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/GPT_as_reviewer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fc79d192167311d"}},{"code_sha256_prefix":"3b835813f4ee9035","entry":"choose_model","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/llama.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b835813f4ee9035"}},{"code_sha256_prefix":"9d842a55a4f2340f","entry":"choose_model","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/mistral.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/mistral.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d842a55a4f2340f"}},{"code_sha256_prefix":"dfb27566a6587fe0","entry":"generate_gemini","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/run2.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/run2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfb27566a6587fe0"}},{"code_sha256_prefix":"a5ae27784e58d31f","entry":"generate_gpt","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/run2.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/run2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5ae27784e58d31f"}},{"code_sha256_prefix":"5279f5e3944b5879","entry":"generate_llama","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/llama.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5279f5e3944b5879"}},{"code_sha256_prefix":"7629eb1d6d7991f4","entry":"generate_llama_70b","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/llama.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7629eb1d6d7991f4"}},{"code_sha256_prefix":"93af4a91787bdbda","entry":"generate_mixtral","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/mistral.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/mistral.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93af4a91787bdbda"}},{"code_sha256_prefix":"ff2defa07993c1f6","entry":"generate_mixtral_8","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/mistral.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/mistral.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff2defa07993c1f6"}},{"code_sha256_prefix":"313d599107b97c5c","entry":"tokenize_and_count","repo":"sandeep82945/future-idea-generation","repo_kind":"official","path":"code/preprocessing.py","file_url":"https://github.com/sandeep82945/future-idea-generation/blob/HEAD/code/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"313d599107b97c5c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}