{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decreasing-annotation-burden-of-pairwise","title":"Decreasing Annotation Burden of Pairwise Comparisons with Human-in-the-Loop Sorting: Application in Medical Image Artifact Rating","arxiv_id":"2202.04823","date":"2022-02-10","proceeding":null,"authors":["Ikbeom Jang","Garrison Danley","Ken Chang","Jayashree Kalpathy-Cramer"],"abstract":"Ranking by pairwise comparisons has shown improved reliability over ordinal classification. However, as the annotations of pairwise comparisons scale quadratically, this becomes less practical when the dataset is large. We propose a method for reducing the number of pairwise comparisons required to rank by a quantitative metric, demonstrating the effectiveness of the approach in ranking medical images by image quality in this proof of concept study. Using the medical image annotation software that we developed, we actively subsample pairwise comparisons using a sorting algorithm with a human rater in the loop. We find that this method substantially reduces the number of comparisons required for a full ordinal ranking without compromising inter-rater reliability when compared to pairwise comparisons without sorting.","url_abs":"https://arxiv.org/abs/2202.04823v1","url_pdf":"https://arxiv.org/pdf/2202.04823v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decreasing-annotation-burden-of-pairwise","repo_url":"https://github.com/gsnlyd/slicelabeler","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Unlicense"}}],"tasks":[{"task_slug":"ordinal-classification","task_name":"Ordinal Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2202.04823","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.04823"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/gsnlyd/SliceLabeler","reach":{"status":"ok","spdx":"Unlicense"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gsnlyd/slicelabeler","reach":{"status":"ok","spdx":"Unlicense"}}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"93d656c4fda4b30b","entry":"inc_dir_name","repo":"gsnlyd/slicelabeler","repo_kind":"official","path":"scripts/export_slices.py","file_url":"https://github.com/gsnlyd/slicelabeler/blob/HEAD/scripts/export_slices.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Unlicense","inline_ok":true,"mcp_get_code":{"code_sha256":"93d656c4fda4b30b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}