{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-14965","title":"Benchmarking Instance-Dependent Label Noise with Controlled Corruptions","arxiv_id":"2606.14965","date":"2026-06-12","proceeding":null,"authors":["Shadman Islam","Agustinus Kristiadi","Mostafa Milani"],"abstract":"Synthetic instance-dependent label noise (IDN) benchmarks are widely used to evaluate noisy-label learning methods, yet existing approaches typically generate noise through imperfect annotators or classifier raters, leaving the source of ambiguity implicit. We introduce CILN, a benchmark generation framework that creates IDN through controlled input corruptions. A diverse voter pool labels corrupted instances, producing benchmark datasets in which both the source and severity of ambiguity are explicit and controllable. Using CIFAR10, MNIST, and Adult, we construct 90 benchmark settings spanning multiple corruption families and severity levels. Our experiments show that the resulting benchmarks exhibit genuine instance-dependent noise, provide diverse confusion structures, and, on CIFAR-10, can produce label distributions that are closer to human uncertainty than an existing synthetic IDN benchmark. We further demonstrate that corruption-mediated IDN can expose failure modes of popular noisy-label learning methods, including Co-Teaching and DivideMix, that are not observed under comparable levels of rater-fallibility noise. These findings suggest that noise structure, not only noise rate, plays an important role in benchmark difficulty and algorithm behavior. By making ambiguity generation explicit and controllable, CILN provides a complementary benchmarking framework for studying noisy-label learning under diverse sources of instance difficulty.","url_abs":"https://arxiv.org/abs/2606.14965","url_pdf":"https://arxiv.org/pdf/2606.14965","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2606.14965","atlas_url":"https://app.syntology.ai/?focus=2606.14965","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.14965"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/sh-islam/ciln-bench","reach":null}],"summary":{"ran":12,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":17,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"16ba1ce730f0f0c4","entry":"apply_image","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/apply.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/apply.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16ba1ce730f0f0c4"}},{"code_sha256_prefix":"8b6577fe86e7c5f0","entry":"bincount_dist","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"code/analyze/canonical/compute_idn_2x2.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/code/analyze/canonical/compute_idn_2x2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b6577fe86e7c5f0"}},{"code_sha256_prefix":"8e9b737a645d1777","entry":"cifar10h_upsampled","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"code/analyze/canonical/tv_vs_cifar10h.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/code/analyze/canonical/tv_vs_cifar10h.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e9b737a645d1777"}},{"code_sha256_prefix":"a30ec2a9c07d8834","entry":"corrupt_image_dataset","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/apply.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/apply.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a30ec2a9c07d8834"}},{"code_sha256_prefix":"6b3b798e0249d295","entry":"corrupt_tabular_dataset","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/apply.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/apply.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b3b798e0249d295"}},{"code_sha256_prefix":"2dfb0af5cd82bfb6","entry":"load_image_dataset","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/validators.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/validators.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2dfb0af5cd82bfb6"}},{"code_sha256_prefix":"33ce7155313bae6b","entry":"load_image_labels","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/validators.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/validators.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33ce7155313bae6b"}},{"code_sha256_prefix":"a88f75234320a417","entry":"m1_rate_variance","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"code/analyze/canonical/compute_idn_2x2.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/code/analyze/canonical/compute_idn_2x2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a88f75234320a417"}},{"code_sha256_prefix":"d229d2c7500bd806","entry":"m2_frobenius","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"code/analyze/canonical/compute_idn_2x2.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/code/analyze/canonical/compute_idn_2x2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d229d2c7500bd806"}},{"code_sha256_prefix":"cea61e9961e1418b","entry":"soft_distribution_from_voters","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"code/analyze/canonical/tv_vs_cifar10h.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/code/analyze/canonical/tv_vs_cifar10h.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cea61e9961e1418b"}},{"code_sha256_prefix":"b0658fbeab6fa061","entry":"summarize_image_dataset","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/validators.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/validators.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0658fbeab6fa061"}},{"code_sha256_prefix":"02e54a70f785ffaa","entry":"symmetric_baseline","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"code/analyze/canonical/tv_vs_cifar10h.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/code/analyze/canonical/tv_vs_cifar10h.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02e54a70f785ffaa"}},{"code_sha256_prefix":"3b6e9699db524864","entry":"ask_choice","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/prompts.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b6e9699db524864"}},{"code_sha256_prefix":"dc322d6e7b70baff","entry":"ask_severity","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/prompts.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc322d6e7b70baff"}},{"code_sha256_prefix":"a5796a02f1661a0c","entry":"ask_text","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/prompts.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5796a02f1661a0c"}},{"code_sha256_prefix":"93085f4555c80b0e","entry":"find_type","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/registry.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93085f4555c80b0e"}},{"code_sha256_prefix":"80b58f851917d179","entry":"get_modality","repo":"sh-islam/ciln-bench","repo_kind":"found_in_text","path":"cli/registry.py","file_url":"https://github.com/sh-islam/ciln-bench/blob/HEAD/cli/registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80b58f851917d179"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}