{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/counterfactual-reasoning-for-multi-label","title":"Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Training","arxiv_id":"2404.06287","date":"2024-04-09","proceeding":null,"authors":["Ming-Kun Xie","Jia-Hao Xiao","Pei Peng","Gang Niu","Masashi Sugiyama","Sheng-Jun Huang"],"abstract":"The key to multi-label image classification (MLC) is to improve model performance by leveraging label correlations. Unfortunately, it has been shown that overemphasizing co-occurrence relationships can cause the overfitting issue of the model, ultimately leading to performance degradation. In this paper, we provide a causal inference framework to show that the correlative features caused by the target object and its co-occurring objects can be regarded as a mediator, which has both positive and negative impacts on model predictions. On the positive side, the mediator enhances the recognition performance of the model by capturing co-occurrence relationships; on the negative side, it has the harmful causal effect that causes the model to make an incorrect prediction for the target object, even when only co-occurring objects are present in an image. To address this problem, we propose a counterfactual reasoning method to measure the total direct effect, achieved by enhancing the direct effect caused only by the target object. Due to the unknown location of the target object, we propose patching-based training and inference to accomplish this goal, which divides an image into multiple patches and identifies the pivot patch that contains the target object. Experimental results on multiple benchmark datasets with diverse configurations validate that the proposed method can achieve state-of-the-art performance.","url_abs":"https://arxiv.org/abs/2404.06287v2","url_pdf":"https://arxiv.org/pdf/2404.06287v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"counterfactual-reasoning-for-multi-label","repo_url":"https://github.com/xiemk/mlc-pat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"causal-inference","task_name":"Causal Inference"},{"task_slug":"counterfactual-reasoning","task_name":"Counterfactual Reasoning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"multi-label-image-classification","task_name":"Multi-Label Image Classification"},{"task_slug":"object","task_name":"Object"},{"task_slug":null,"task_name":"counterfactual"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"causal-inference","method_name":"Causal inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.06287","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.06287"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiemk/mlc-pat","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran":8,"ran_draft_wrong":2,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":15,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"f52b226226096fdd","entry":"average_precision","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/helper_functions/helper_functions.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/helper_functions/helper_functions.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f52b226226096fdd"}},{"code_sha256_prefix":"c1a4354520c04ab6","entry":"calculate_metric","repo":"xiemk/mlc-pat","repo_kind":"official","path":"run_PAT-I.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/run_PAT-I.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1a4354520c04ab6"}},{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/models/resnet.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/models/resnet.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/models/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"c324d082f762772c","entry":"get_model_path","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/models/factory.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/models/factory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c324d082f762772c"}},{"code_sha256_prefix":"96e62b62824b856f","entry":"get_raw_dict","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/helper_functions/helper_functions.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/helper_functions/helper_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96e62b62824b856f"}},{"code_sha256_prefix":"6840401a1f3d4796","entry":"kill_process","repo":"xiemk/mlc-pat","repo_kind":"official","path":"run_PAT-T.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/run_PAT-T.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6840401a1f3d4796"}},{"code_sha256_prefix":"bc7c582ef8068c42","entry":"linear_head","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/models/factory.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/models/factory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc7c582ef8068c42"}},{"code_sha256_prefix":"044fe3f67aac5e39","entry":"parse_args","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/helper_functions/helper_functions.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/helper_functions/helper_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"044fe3f67aac5e39"}},{"code_sha256_prefix":"ec3bc56d56e88c4b","entry":"resnet","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/utils/model.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec3bc56d56e88c4b"}},{"code_sha256_prefix":"5f1d5c1452000082","entry":"thresholding","repo":"xiemk/mlc-pat","repo_kind":"official","path":"run_PAT-I.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/run_PAT-I.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f1d5c1452000082"}},{"code_sha256_prefix":"fa1eb87122247e2f","entry":"to_ddp","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/helper_functions/distributed.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/helper_functions/distributed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa1eb87122247e2f"}},{"code_sha256_prefix":"dffe6300676b187f","entry":"create_model","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/utils/model.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/utils/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dffe6300676b187f"}},{"code_sha256_prefix":"e68752297a7b78de","entry":"q2l_head","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/models/factory.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/models/factory.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e68752297a7b78de"}},{"code_sha256_prefix":"14265d790751cfed","entry":"reduce_tensor","repo":"xiemk/mlc-pat","repo_kind":"official","path":"src_files/helper_functions/distributed.py","file_url":"https://github.com/xiemk/mlc-pat/blob/HEAD/src_files/helper_functions/distributed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14265d790751cfed"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}