{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/designing-an-evaluation-framework-for-large","title":"Designing an Evaluation Framework for Large Language Models in Astronomy Research","arxiv_id":"2405.20389","date":"2024-05-30","proceeding":null,"authors":["John F. Wu","Alina Hyk","Kiera McCormick","Christine Ye","Simone Astarita","Elina Baral","Jo Ciuca","Jesse Cranney","Anjalie Field","Kartheik Iyer","Philipp Koehn","Jenn Kotler","Sandor Kruk","Michelle Ntampaka","Charles O'Neill","Joshua E. G. Peek","Sanjib Sharma","Mikaeel Yunus"],"abstract":"Large Language Models (LLMs) are shifting how scientific research is done. It is imperative to understand how researchers interact with these models and how scientific sub-communities like astronomy might benefit from them. However, there is currently no standard for evaluating the use of LLMs in astronomy. Therefore, we present the experimental design for an evaluation study on how astronomy researchers interact with LLMs. We deploy a Slack chatbot that can answer queries from users via Retrieval-Augmented Generation (RAG); these responses are grounded in astronomy papers from arXiv. We record and anonymize user questions and chatbot answers, user upvotes and downvotes to LLM responses, user feedback to the LLM, and retrieved documents and similarity scores with the query. Our data collection method will enable future dynamic evaluations of LLM tools for astronomy.","url_abs":"https://arxiv.org/abs/2405.20389v1","url_pdf":"https://arxiv.org/pdf/2405.20389v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"designing-an-evaluation-framework-for-large","repo_url":"https://github.com/jsalt2024-evaluating-llms-for-astronomy/astro-arxiv-bot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"astronomy","task_name":"Astronomy"},{"task_slug":"chatbot","task_name":"Chatbot"},{"task_slug":"experimental-design","task_name":"Experimental Design"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.20389","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.20389"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jsalt2024-evaluating-llms-for-astronomy/astro-arxiv-bot","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6ef49315981b1e8f","entry":"make_arxiv_link","repo":"jsalt2024-evaluating-llms-for-astronomy/astro-arxiv-bot","repo_kind":"official","path":"src/llm-slackbot.py","file_url":"https://github.com/jsalt2024-evaluating-llms-for-astronomy/astro-arxiv-bot/blob/HEAD/src/llm-slackbot.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ef49315981b1e8f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}