{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-reviewers-lock-horn-finding-disagreement","title":"When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews","arxiv_id":"2310.18685","date":"2023-10-28","proceeding":null,"authors":["Sandeep Kumar","Tirthankar Ghosal","Asif Ekbal"],"abstract":"To this date, the efficacy of the scientific publishing enterprise fundamentally rests on the strength of the peer review process. The journal editor or the conference chair primarily relies on the expert reviewers' assessment, identify points of agreement and disagreement and try to reach a consensus to make a fair and informed decision on whether to accept or reject a paper. However, with the escalating number of submissions requiring review, especially in top-tier Artificial Intelligence (AI) conferences, the editor/chair, among many other works, invests a significant, sometimes stressful effort to mitigate reviewer disagreements. Here in this work, we introduce a novel task of automatically identifying contradictions among reviewers on a given article. To this end, we introduce ContraSciView, a comprehensive review-pair contradiction dataset on around 8.5k papers (with around 28k review pairs containing nearly 50k review pair comments) from the open review-based ICLR and NeurIPS conferences. We further propose a baseline model that detects contradictory statements from the review pairs. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically. We make our dataset and code public for further investigations.","url_abs":"https://arxiv.org/abs/2310.18685v1","url_pdf":"https://arxiv.org/pdf/2310.18685v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-reviewers-lock-horn-finding-disagreement","repo_url":"https://github.com/sandeep82945/contradiction-in-peer-review","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2310.18685","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.18685"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/dsdanielpark/Bard-API","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sandeep82945/contradiction-in-peer-review","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0c0742a4ea1d9582","entry":"append_item_from_dict_to_list","repo":"sandeep82945/contradiction-in-peer-review","repo_kind":"official","path":"src/utils/list_dict_data_tool.py","file_url":"https://github.com/sandeep82945/contradiction-in-peer-review/blob/HEAD/src/utils/list_dict_data_tool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c0742a4ea1d9582"}},{"code_sha256_prefix":"8379fdb8173a8157","entry":"calculate_metrics","repo":"sandeep82945/contradiction-in-peer-review","repo_kind":"official","path":"src/training_scratch3.py","file_url":"https://github.com/sandeep82945/contradiction-in-peer-review/blob/HEAD/src/training_scratch3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8379fdb8173a8157"}},{"code_sha256_prefix":"96627f234d4de785","entry":"count_F1_manual","repo":"sandeep82945/contradiction-in-peer-review","repo_kind":"official","path":"src/training_scratch3.py","file_url":"https://github.com/sandeep82945/contradiction-in-peer-review/blob/HEAD/src/training_scratch3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"96627f234d4de785"}},{"code_sha256_prefix":"5c0d2b710a43c732","entry":"dict_to_list","repo":"sandeep82945/contradiction-in-peer-review","repo_kind":"official","path":"src/utils/list_dict_data_tool.py","file_url":"https://github.com/sandeep82945/contradiction-in-peer-review/blob/HEAD/src/utils/list_dict_data_tool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c0d2b710a43c732"}},{"code_sha256_prefix":"c71aafcb215c5122","entry":"has_tensor","repo":"sandeep82945/contradiction-in-peer-review","repo_kind":"official","path":"src/flint/data_utils/batchbuilder.py","file_url":"https://github.com/sandeep82945/contradiction-in-peer-review/blob/HEAD/src/flint/data_utils/batchbuilder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c71aafcb215c5122"}},{"code_sha256_prefix":"66cbfd6d3a76a33d","entry":"list_to_dict","repo":"sandeep82945/contradiction-in-peer-review","repo_kind":"official","path":"src/utils/list_dict_data_tool.py","file_url":"https://github.com/sandeep82945/contradiction-in-peer-review/blob/HEAD/src/utils/list_dict_data_tool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66cbfd6d3a76a33d"}},{"code_sha256_prefix":"7ebeb1ae8084fb58","entry":"move_to_device","repo":"sandeep82945/contradiction-in-peer-review","repo_kind":"official","path":"src/flint/data_utils/batchbuilder.py","file_url":"https://github.com/sandeep82945/contradiction-in-peer-review/blob/HEAD/src/flint/data_utils/batchbuilder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7ebeb1ae8084fb58"}},{"code_sha256_prefix":"0711b3689395f3a7","entry":"json_dumps","repo":"sandeep82945/contradiction-in-peer-review","repo_kind":"official","path":"src/utils/common.py","file_url":"https://github.com/sandeep82945/contradiction-in-peer-review/blob/HEAD/src/utils/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0711b3689395f3a7"}},{"code_sha256_prefix":"821813f4d585524f","entry":"json_loads","repo":"sandeep82945/contradiction-in-peer-review","repo_kind":"official","path":"src/utils/common.py","file_url":"https://github.com/sandeep82945/contradiction-in-peer-review/blob/HEAD/src/utils/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"821813f4d585524f"}},{"code_sha256_prefix":"f0ff1b43a0f5291d","entry":"unserialize_JsonableObject","repo":"sandeep82945/contradiction-in-peer-review","repo_kind":"official","path":"src/utils/common.py","file_url":"https://github.com/sandeep82945/contradiction-in-peer-review/blob/HEAD/src/utils/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0ff1b43a0f5291d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}