{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/modeldiff-a-framework-for-comparing-learning","title":"ModelDiff: A Framework for Comparing Learning Algorithms","arxiv_id":"2211.12491","date":"2022-11-22","proceeding":null,"authors":["Harshay Shah","Sung Min Park","Andrew Ilyas","Aleksander Madry"],"abstract":"We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms. We begin by formalizing this goal as one of finding distinguishing feature transformations, i.e., input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present ModelDiff, a method that leverages the datamodels framework (Ilyas et al., 2022) to compare learning algorithms based on how they use their training data. We demonstrate ModelDiff through three case studies, comparing models trained with/without data augmentation, with/without pre-training, and with different SGD hyperparameters. Our code is available at https://github.com/MadryLab/modeldiff .","url_abs":"https://arxiv.org/abs/2211.12491v1","url_pdf":"https://arxiv.org/pdf/2211.12491v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"modeldiff-a-framework-for-comparing-learning","repo_url":"https://github.com/madrylab/modeldiff","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.12491","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.12491"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/madrylab/modeldiff","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"acf9b3d8d24c36de","entry":"get_model","repo":"madrylab/modeldiff","repo_kind":"official","path":"counterfactuals/eval_cifar_blackwhite.py","file_url":"https://github.com/madrylab/modeldiff/blob/HEAD/counterfactuals/eval_cifar_blackwhite.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acf9b3d8d24c36de"}},{"code_sha256_prefix":"8ea06aa985e30d40","entry":"get_model","repo":"madrylab/modeldiff","repo_kind":"official","path":"counterfactuals/eval_living17_polkadots.py","file_url":"https://github.com/madrylab/modeldiff/blob/HEAD/counterfactuals/eval_living17_polkadots.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ea06aa985e30d40"}},{"code_sha256_prefix":"ac04d374ee116819","entry":"get_model","repo":"madrylab/modeldiff","repo_kind":"official","path":"counterfactuals/eval_waterbirds_humanface.py","file_url":"https://github.com/madrylab/modeldiff/blob/HEAD/counterfactuals/eval_waterbirds_humanface.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac04d374ee116819"}},{"code_sha256_prefix":"5f2b6f178e47ba0a","entry":"get_model","repo":"madrylab/modeldiff","repo_kind":"official","path":"counterfactuals/eval_waterbirds_yellowcolor.py","file_url":"https://github.com/madrylab/modeldiff/blob/HEAD/counterfactuals/eval_waterbirds_yellowcolor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f2b6f178e47ba0a"}},{"code_sha256_prefix":"999976313130caee","entry":"get_residual_attributions","repo":"madrylab/modeldiff","repo_kind":"official","path":"modeldiff/pca.py","file_url":"https://github.com/madrylab/modeldiff/blob/HEAD/modeldiff/pca.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"999976313130caee"}},{"code_sha256_prefix":"ad183272834a0d18","entry":"get_thresholded_image","repo":"madrylab/modeldiff","repo_kind":"official","path":"counterfactuals/eval_living17_spiderweb.py","file_url":"https://github.com/madrylab/modeldiff/blob/HEAD/counterfactuals/eval_living17_spiderweb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad183272834a0d18"}},{"code_sha256_prefix":"e99be7f1664ac47f","entry":"get_variance_along_directions","repo":"madrylab/modeldiff","repo_kind":"official","path":"modeldiff/pca.py","file_url":"https://github.com/madrylab/modeldiff/blob/HEAD/modeldiff/pca.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e99be7f1664ac47f"}},{"code_sha256_prefix":"941e064be06d7c71","entry":"residual_pca","repo":"madrylab/modeldiff","repo_kind":"official","path":"modeldiff/pca.py","file_url":"https://github.com/madrylab/modeldiff/blob/HEAD/modeldiff/pca.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"941e064be06d7c71"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}