{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/are-you-getting-what-you-pay-for-auditing","title":"Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs","arxiv_id":"2504.04715","date":"2025-04-07","proceeding":null,"authors":["Will Cai","Tianneng Shi","Xuandong Zhao","Dawn Song"],"abstract":"The proliferation of Large Language Models (LLMs) accessed via black-box APIs introduces a significant trust challenge: users pay for services based on advertised model capabilities (e.g., size, performance), but providers may covertly substitute the specified model with a cheaper, lower-quality alternative to reduce operational costs. This lack of transparency undermines fairness, erodes trust, and complicates reliable benchmarking. Detecting such substitutions is difficult due to the black-box nature, typically limiting interaction to input-output queries. This paper formalizes the problem of model substitution detection in LLM APIs. We systematically evaluate existing verification techniques, including output-based statistical tests, benchmark evaluations, and log probability analysis, under various realistic attack scenarios like model quantization, randomized substitution, and benchmark evasion. Our findings reveal the limitations of methods relying solely on text outputs, especially against subtle or adaptive attacks. While log probability analysis offers stronger guarantees when available, its accessibility is often limited. We conclude by discussing the potential of hardware-based solutions like Trusted Execution Environments (TEEs) as a pathway towards provable model integrity, highlighting the trade-offs between security, performance, and provider adoption. Code is available at https://github.com/sunblaze-ucb/llm-api-audit","url_abs":"https://arxiv.org/abs/2504.04715v1","url_pdf":"https://arxiv.org/pdf/2504.04715v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"are-you-getting-what-you-pay-for-auditing","repo_url":"https://github.com/sunblaze-ucb/llm-api-audit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"fairness","task_name":"Fairness"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2504.04715","atlas_url":"https://app.syntology.ai/?focus=2504.04715","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.04715"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sunblaze-ucb/llm-api-audit","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"20a7cc804eb22661","entry":"hash_args","repo":"sunblaze-ucb/llm-api-audit","repo_kind":"official","path":"benchmark/lm-evaluation-harness/lm_eval/api/model.py","file_url":"https://github.com/sunblaze-ucb/llm-api-audit/blob/HEAD/benchmark/lm-evaluation-harness/lm_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20a7cc804eb22661"}},{"code_sha256_prefix":"af00e66a13fa87a7","entry":"policy","repo":"sunblaze-ucb/llm-api-audit","repo_kind":"official","path":"model_equality_testing/experiments/sampling/cache_api_samples.py","file_url":"https://github.com/sunblaze-ucb/llm-api-audit/blob/HEAD/model_equality_testing/experiments/sampling/cache_api_samples.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af00e66a13fa87a7"}},{"code_sha256_prefix":"894f96b26abd7ab3","entry":"get_model","repo":"sunblaze-ucb/llm-api-audit","repo_kind":"official","path":"benchmark/lm-evaluation-harness/lm_eval/api/registry.py","file_url":"https://github.com/sunblaze-ucb/llm-api-audit/blob/HEAD/benchmark/lm-evaluation-harness/lm_eval/api/registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"894f96b26abd7ab3"}},{"code_sha256_prefix":"d3ee5f783a8a7d3f","entry":"get_sampler","repo":"sunblaze-ucb/llm-api-audit","repo_kind":"official","path":"benchmark/lm-evaluation-harness/lm_eval/api/samplers.py","file_url":"https://github.com/sunblaze-ucb/llm-api-audit/blob/HEAD/benchmark/lm-evaluation-harness/lm_eval/api/samplers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3ee5f783a8a7d3f"}},{"code_sha256_prefix":"1bbf582194532683","entry":"register_group","repo":"sunblaze-ucb/llm-api-audit","repo_kind":"official","path":"benchmark/lm-evaluation-harness/lm_eval/api/registry.py","file_url":"https://github.com/sunblaze-ucb/llm-api-audit/blob/HEAD/benchmark/lm-evaluation-harness/lm_eval/api/registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bbf582194532683"}},{"code_sha256_prefix":"6a4a70c8a645fde8","entry":"register_task","repo":"sunblaze-ucb/llm-api-audit","repo_kind":"official","path":"benchmark/lm-evaluation-harness/lm_eval/api/registry.py","file_url":"https://github.com/sunblaze-ucb/llm-api-audit/blob/HEAD/benchmark/lm-evaluation-harness/lm_eval/api/registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a4a70c8a645fde8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}