{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tripjudge-a-relevance-judgement-test","title":"TripJudge: A Relevance Judgement Test Collection for TripClick Health Retrieval","arxiv_id":"2208.06936","date":"2022-08-14","proceeding":null,"authors":["Sophia Althammer","Sebastian Hofstätter","Suzan Verberne","Allan Hanbury"],"abstract":"Robust test collections are crucial for Information Retrieval research. Recently there is a growing interest in evaluating retrieval systems for domain-specific retrieval tasks, however these tasks often lack a reliable test collection with human-annotated relevance assessments following the Cranfield paradigm. In the medical domain, the TripClick collection was recently proposed, which contains click log data from the Trip search engine and includes two click-based test sets. However the clicks are biased to the retrieval model used, which remains unknown, and a previous study shows that the test sets have a low judgement coverage for the Top-10 results of lexical and neural retrieval models. In this paper we present the novel, relevance judgement test collection TripJudge for TripClick health retrieval. We collect relevance judgements in an annotation campaign and ensure the quality and reusability of TripJudge by a variety of ranking methods for pool creation, by multiple judgements per query-document pair and by an at least moderate inter-annotator agreement. We compare system evaluation with TripJudge and TripClick and find that that click and judgement-based evaluation can lead to substantially different system rankings.","url_abs":"https://arxiv.org/abs/2208.06936v1","url_pdf":"https://arxiv.org/pdf/2208.06936v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tripjudge-a-relevance-judgement-test","repo_url":"https://github.com/sophiaalthammer/tripjudge","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.06936","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.06936"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sophiaalthammer/tripjudge","reach":null}],"summary":{"ran_draft_wrong":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"6c45e11da88a8dbb","entry":"get_pairs_from_run","repo":"sophiaalthammer/tripjudge","repo_kind":"official","path":"preprocessing/trip_preprocessing.py","file_url":"https://github.com/sophiaalthammer/tripjudge/blob/HEAD/preprocessing/trip_preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c45e11da88a8dbb"}},{"code_sha256_prefix":"76b90930091da8c7","entry":"heuristic_majority_voting","repo":"sophiaalthammer/tripjudge","repo_kind":"official","path":"annotation/generate_qrels.py","file_url":"https://github.com/sophiaalthammer/tripjudge/blob/HEAD/annotation/generate_qrels.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76b90930091da8c7"}},{"code_sha256_prefix":"606f3f1a4b130f22","entry":"read_in_queries","repo":"sophiaalthammer/tripjudge","repo_kind":"official","path":"preprocessing/trip_preprocessing.py","file_url":"https://github.com/sophiaalthammer/tripjudge/blob/HEAD/preprocessing/trip_preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"606f3f1a4b130f22"}},{"code_sha256_prefix":"3b7a9bd762a47b6b","entry":"run_intersection","repo":"sophiaalthammer/tripjudge","repo_kind":"official","path":"preprocessing/trip_preprocessing.py","file_url":"https://github.com/sophiaalthammer/tripjudge/blob/HEAD/preprocessing/trip_preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b7a9bd762a47b6b"}},{"code_sha256_prefix":"22bf72a21e3bd34e","entry":"heuristic_majority_voting2class","repo":"sophiaalthammer/tripjudge","repo_kind":"official","path":"annotation/generate_qrels.py","file_url":"https://github.com/sophiaalthammer/tripjudge/blob/HEAD/annotation/generate_qrels.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22bf72a21e3bd34e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}