{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/chaosmining-a-benchmark-to-evaluate-post-hoc","title":"ChaosMining: A Benchmark to Evaluate Post-Hoc Local Attribution Methods in Low SNR Environments","arxiv_id":"2406.12150","date":"2024-06-17","proceeding":null,"authors":["Ge Shi","Ziwen Kan","Jason Smucny","Ian Davidson"],"abstract":"In this study, we examine the efficacy of post-hoc local attribution methods in identifying features with predictive power from irrelevant ones in domains characterized by a low signal-to-noise ratio (SNR), a common scenario in real-world machine learning applications. We developed synthetic datasets encompassing symbolic functional, image, and audio data, incorporating a benchmark on the {\\it (Model \\(\\times\\) Attribution\\(\\times\\) Noise Condition)} triplet. By rigorously testing various classic models trained from scratch, we gained valuable insights into the performance of these attribution methods in multiple conditions. Based on these findings, we introduce a novel extension to the notable recursive feature elimination (RFE) algorithm, enhancing its applicability for neural networks. Our experiments highlight its strengths in prediction and feature selection, alongside limitations in scalability. Further details and additional minor findings are included in the appendix, with extensive discussions. The codes and resources are available at \\href{https://github.com/geshijoker/ChaosMining/}{URL}.","url_abs":"https://arxiv.org/abs/2406.12150v1","url_pdf":"https://arxiv.org/pdf/2406.12150v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"chaosmining-a-benchmark-to-evaluate-post-hoc","repo_url":"https://github.com/geshijoker/chaosmining","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":null,"task_name":"Triplet"},{"task_slug":"feature-selection","task_name":"feature selection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.12150","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12150"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/geshijoker/chaosmining","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":9,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"3f2d8ff2523ae6bb","entry":"abs_argmax_topk","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/simulation/functions.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/simulation/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3f2d8ff2523ae6bb"}},{"code_sha256_prefix":"2bc8839905eaed0a","entry":"calculate_multi_iou","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/vision/contribs.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/vision/contribs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2bc8839905eaed0a"}},{"code_sha256_prefix":"48b5e8bb19816873","entry":"check_make_dir","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/utils.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"48b5e8bb19816873"}},{"code_sha256_prefix":"f00860b5413118cf","entry":"collate_fn","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/audio/functions.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/audio/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f00860b5413118cf"}},{"code_sha256_prefix":"2a80220dabcb742a","entry":"conv1x1","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/vision/models.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/vision/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a80220dabcb742a"}},{"code_sha256_prefix":"600ff2c45e0de056","entry":"conv3x3","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/vision/models.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/vision/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"600ff2c45e0de056"}},{"code_sha256_prefix":"4a0fe6e0d62d1fef","entry":"get_multi_box","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/vision/contribs.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/vision/contribs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4a0fe6e0d62d1fef"}},{"code_sha256_prefix":"ff44532ba90692b7","entry":"mean_std_absolute_error","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/simulation/functions.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/simulation/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ff44532ba90692b7"}},{"code_sha256_prefix":"711f9fea58b1efb2","entry":"mean_std_absolute_error_ratio","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/simulation/functions.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/simulation/functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"711f9fea58b1efb2"}},{"code_sha256_prefix":"6dd50a63b04bbb7f","entry":"radar_factory","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/utils.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6dd50a63b04bbb7f"}},{"code_sha256_prefix":"b86d51455d3ce320","entry":"read_formulas","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/data_utils.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b86d51455d3ce320"}},{"code_sha256_prefix":"19f316a66070d0d9","entry":"create_simulation_data","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/data_utils.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"19f316a66070d0d9"}},{"code_sha256_prefix":"4ec1566ded61e473","entry":"index_to_label","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/audio/functions.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/audio/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4ec1566ded61e473"}},{"code_sha256_prefix":"b297bc22d7505f05","entry":"label_to_index","repo":"geshijoker/chaosmining","repo_kind":"official","path":"chaosmining/audio/functions.py","file_url":"https://github.com/geshijoker/chaosmining/blob/HEAD/chaosmining/audio/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b297bc22d7505f05"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}