{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-equality-testing-which-model-is-this","title":"Model Equality Testing: Which Model Is This API Serving?","arxiv_id":"2410.20247","date":"2024-10-26","proceeding":null,"authors":["Irena Gao","Percy Liang","Carlos Guestrin"],"abstract":"Users often interact with large language models through black-box inference APIs, both for closed- and open-weight models (e.g., Llama models are popularly accessed via Amazon Bedrock and Azure AI Studio). In order to cut costs or add functionality, API providers may quantize, watermark, or finetune the underlying model, changing the output distribution -- often without notifying users. We formalize detecting such distortions as Model Equality Testing, a two-sample testing problem, where the user collects samples from the API and a reference distribution and conducts a statistical test to see if the two distributions are the same. We find that tests based on the Maximum Mean Discrepancy between distributions are powerful for this task: a test built on a simple string kernel achieves a median of 77.4% power against a range of distortions, using an average of just 10 samples per prompt. We then apply this test to commercial inference APIs for four Llama models, finding that 11 out of 31 endpoints serve different distributions than reference weights released by Meta.","url_abs":"https://arxiv.org/abs/2410.20247v1","url_pdf":"https://arxiv.org/pdf/2410.20247v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-equality-testing-which-model-is-this","repo_url":"https://github.com/i-gao/model-equality-testing","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"hypothesis-testing","task_name":"Two-sample testing"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.20247","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.20247"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/i-gao/model-equality-testing","reach":{"status":"ok"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"ac1b7872a50ce89b","entry":"is_pretrained_ckpt","repo":"i-gao/model-equality-testing","repo_kind":"official","path":"experiments/sampling/model.py","file_url":"https://github.com/i-gao/model-equality-testing/blob/HEAD/experiments/sampling/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac1b7872a50ce89b"}},{"code_sha256_prefix":"43df07c2bee912a3","entry":"load_parametric_bootstrap","repo":"i-gao/model-equality-testing","repo_kind":"official","path":"experiments/testing/bootstrap_manager.py","file_url":"https://github.com/i-gao/model-equality-testing/blob/HEAD/experiments/testing/bootstrap_manager.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"43df07c2bee912a3"}},{"code_sha256_prefix":"4d86fb786ab0c1dd","entry":"pad_to_length","repo":"i-gao/model-equality-testing","repo_kind":"official","path":"model_equality_testing/src/utils.py","file_url":"https://github.com/i-gao/model-equality-testing/blob/HEAD/model_equality_testing/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d86fb786ab0c1dd"}},{"code_sha256_prefix":"ee41f9a8c5aa9ab8","entry":"sanitize","repo":"i-gao/model-equality-testing","repo_kind":"official","path":"model_equality_testing/src/utils.py","file_url":"https://github.com/i-gao/model-equality-testing/blob/HEAD/model_equality_testing/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee41f9a8c5aa9ab8"}},{"code_sha256_prefix":"980f947e9e845d58","entry":"tokenize_unicode","repo":"i-gao/model-equality-testing","repo_kind":"official","path":"model_equality_testing/src/utils.py","file_url":"https://github.com/i-gao/model-equality-testing/blob/HEAD/model_equality_testing/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"980f947e9e845d58"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}