{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llm-assisted-relevance-assessments-when","title":"LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help?","arxiv_id":"2411.06877","date":"2024-11-11","proceeding":null,"authors":["Rikiya Takehi","Ellen M. Voorhees","Tetsuya Sakai","Ian Soboroff"],"abstract":"Test collections are information retrieval tools that allow researchers to quickly and easily evaluate ranking algorithms. While test collections have become an integral part of IR research, the process of data creation involves significant effort in manual annotations, which often makes it very expensive and time-consuming. Thus, test collections could become too small when the budget is limited, which may lead to unstable evaluations. As a cheaper alternative, recent studies have proposed the use of large language models (LLMs) to completely replace human assessors. However, while LLMs seem to somewhat correlate with human judgments, their predictions are not perfect and often show bias. Thus a complete replacement with LLMs is argued to be too risky and not fully reliable. Thus, in this paper, we propose LLM-Assisted Relevance Assessments (LARA), an effective method to balance manual annotations with LLM annotations, which helps to build a rich and reliable test collection even under a low budget. We use the LLM's predicted relevance probabilities to select the most profitable documents to manually annotate under a budget constraint. With theoretical reasoning, LARA effectively guides the human annotation process by actively learning to calibrate the LLM's predicted relevance probabilities. Then, using the calibration model learned from the limited manual annotations, LARA debiases the LLM predictions to annotate the remaining non-assessed data. Empirical evaluations on TREC-7 Ad Hoc, TREC-8 Ad Hoc, TREC Robust 2004, and TREC-COVID datasets show that LARA outperforms alternative solutions under almost any budget constraint.","url_abs":"https://arxiv.org/abs/2411.06877v3","url_pdf":"https://arxiv.org/pdf/2411.06877v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llm-assisted-relevance-assessments-when","repo_url":"https://github.com/RikiyaT/LARA","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"}],"methods":[{"method_slug":"hoc","method_name":"HOC"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.06877","atlas_url":"https://app.syntology.ai/?focus=2411.06877","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.06877"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RikiyaT/LARA","reach":null}],"summary":{"ran_draft_wrong":4,"ran_fixture":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"50b8620ec9dd24c0","entry":"get_document","repo":"RikiyaT/LARA","repo_kind":"official","path":"src/example-covid-itachi.py","file_url":"https://github.com/RikiyaT/LARA/blob/HEAD/src/example-covid-itachi.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50b8620ec9dd24c0"}},{"code_sha256_prefix":"7dfeae6eea40e023","entry":"load_previous_results","repo":"RikiyaT/LARA","repo_kind":"official","path":"src/test-al.py","file_url":"https://github.com/RikiyaT/LARA/blob/HEAD/src/test-al.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7dfeae6eea40e023"}},{"code_sha256_prefix":"2764249ced7bbd4f","entry":"load_qrels","repo":"RikiyaT/LARA","repo_kind":"official","path":"src/example-covid-itachi.py","file_url":"https://github.com/RikiyaT/LARA/blob/HEAD/src/example-covid-itachi.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2764249ced7bbd4f"}},{"code_sha256_prefix":"84b7a5beacbd0e8f","entry":"load_topics","repo":"RikiyaT/LARA","repo_kind":"official","path":"src/example-covid-itachi.py","file_url":"https://github.com/RikiyaT/LARA/blob/HEAD/src/example-covid-itachi.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84b7a5beacbd0e8f"}},{"code_sha256_prefix":"6739fc845a6044d4","entry":"smooth_normalize","repo":"RikiyaT/LARA","repo_kind":"official","path":"src/test-al.py","file_url":"https://github.com/RikiyaT/LARA/blob/HEAD/src/test-al.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6739fc845a6044d4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}