{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/online-selective-classification-with-limited","title":"Online Selective Classification with Limited Feedback","arxiv_id":"2110.14243","date":"2021-10-27","proceeding":"NeurIPS 2021 12","authors":["Aditya Gangrade","Anil Kag","Ashok Cutkosky","Venkatesh Saligrama"],"abstract":"Motivated by applications to resource-limited and safety-critical domains, we study selective classification in the online learning model, wherein a predictor may abstain from classifying an instance. For example, this may model an adaptive decision to invoke more resources on this instance. Two salient aspects of the setting we consider are that the data may be non-realisable, due to which abstention may be a valid long-term action, and that feedback is only received when the learner abstains, which models the fact that reliable labels are only available when the resource intensive processing is invoked. Within this framework, we explore strategies that make few mistakes, while not abstaining too many times more than the best-in-hindsight error-free classifier from a given class. That is, the one that makes no mistakes, while abstaining the fewest number of times. We construct simple versioning-based schemes for any $\\mu \\in (0,1],$ that make most $T^\\mu$ mistakes while incurring \\smash{$\\tilde{O}(T^{1-\\mu})$} excess abstention against adaptive adversaries. We further show that this dependence on $T$ is tight, and provide illustrative experiments on realistic datasets.","url_abs":"https://arxiv.org/abs/2110.14243v1","url_pdf":"https://arxiv.org/pdf/2110.14243v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"online-selective-classification-with-limited","repo_url":"https://github.com/anilkagak2/online-selective-classification","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.14243","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.14243"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/anilkagak2/online-selective-classification","reach":null}],"summary":{"ran_violates":1,"ran_draft_wrong":2,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"7524170926f17cad","entry":"bernoulli_flip","repo":"anilkagak2/online-selective-classification","repo_kind":"official","path":"main_runner_varying_Ts_experiment.py","file_url":"https://github.com/anilkagak2/online-selective-classification/blob/HEAD/main_runner_varying_Ts_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7524170926f17cad"}},{"code_sha256_prefix":"87c1c815b1b49f03","entry":"get_key","repo":"anilkagak2/online-selective-classification","repo_kind":"official","path":"main_runner_varying_prob_p_theta_experiment.py","file_url":"https://github.com/anilkagak2/online-selective-classification/blob/HEAD/main_runner_varying_prob_p_theta_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87c1c815b1b49f03"}},{"code_sha256_prefix":"de36e1aa7fec28b6","entry":"get_predictions_for_all_experts","repo":"anilkagak2/online-selective-classification","repo_kind":"official","path":"main_runner_varying_Ts_experiment.py","file_url":"https://github.com/anilkagak2/online-selective-classification/blob/HEAD/main_runner_varying_Ts_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de36e1aa7fec28b6"}},{"code_sha256_prefix":"8edcf3c2770e6c77","entry":"run_one_experiment","repo":"anilkagak2/online-selective-classification","repo_kind":"official","path":"main_runner_varying_Ts_experiment.py","file_url":"https://github.com/anilkagak2/online-selective-classification/blob/HEAD/main_runner_varying_Ts_experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8edcf3c2770e6c77"}},{"code_sha256_prefix":"57df6ae1cf413219","entry":"find_experts_at_particular_mistakes","repo":"anilkagak2/online-selective-classification","repo_kind":"official","path":"main_runner_varying_prob_p_theta_experiment.py","file_url":"https://github.com/anilkagak2/online-selective-classification/blob/HEAD/main_runner_varying_prob_p_theta_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57df6ae1cf413219"}},{"code_sha256_prefix":"57fff370b52f79c6","entry":"simulate_experts_at_time_T","repo":"anilkagak2/online-selective-classification","repo_kind":"official","path":"main_runner_varying_prob_p_theta_experiment.py","file_url":"https://github.com/anilkagak2/online-selective-classification/blob/HEAD/main_runner_varying_prob_p_theta_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57fff370b52f79c6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}