{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-31108","title":"Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions","arxiv_id":"2608.31108","date":"2026-08-31","proceeding":null,"authors":["Ahmed El Kady","Aravind Narayanan","Rehana Noorani","Yani Ioannou","Shaina Raza"],"abstract":"Efficient evaluation changes the protocol used to support claims about model behavior, yet it is rarely tested whether those claims remain stable after the evaluation itself is made cheaper. We stress-test conclusion robustness in responsible-AI benchmarking by evaluating three dense and mixture-of-experts models on BBQ and BBQ-V under seven conditions spanning batching, quantization, benchmark reduction, and their combinations. Rather than treating preserved aggregate accuracy as sufficient, we compare accuracy, bias severity and prevalence, reasoning quality, subgroup behavior, subset-membership stability, runtime, and measured GPU energy against a full-benchmark BF16 baseline. Larger batching keeps accuracy within 0.35 percentage points of baseline and produces comparatively small subgroup changes, while reducing energy in five of six model--dataset settings. INT8 largely preserves quality but uses 1.79--4.26$\\times$ baseline energy. INT4 causes larger, model- and context-dependent changes. Reduced benchmarks provide the most consistent savings, but very small subsets are substantially more sensitive to which items are retained. Efficient evaluation should therefore be treated as a measurement intervention whose validity must be checked across the conclusions the benchmark is intended to support. Our project website is https://vectorinstitute.github.io/sustainable-rai-evaluation/ and the code is available at https://github.com/VectorInstitute/sustainable-rai-evaluation.","url_abs":"https://arxiv.org/abs/2608.31108","url_pdf":"https://arxiv.org/pdf/2608.31108","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2608.31108","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.31108"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/VectorInstitute/sustainable-rai-evaluation","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":19},"by_repo_kind":{"found_in_text":{"samples":19,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4bcdfcee5a21382a","entry":"bias_present","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/prompts.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4bcdfcee5a21382a"}},{"code_sha256_prefix":"0faada8059bc5ec4","entry":"breakdown","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/metrics.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0faada8059bc5ec4"}},{"code_sha256_prefix":"6e2c6a1f63549198","entry":"build_manifest","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/subsets.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/subsets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e2c6a1f63549198"}},{"code_sha256_prefix":"f1880abc92187acb","entry":"build_messages","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/inference.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f1880abc92187acb"}},{"code_sha256_prefix":"c5b203c994116675","entry":"carbon_kg","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/telemetry.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/telemetry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c5b203c994116675"}},{"code_sha256_prefix":"849e1e708aeb084e","entry":"dataset_fingerprint","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/datasets.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"849e1e708aeb084e"}},{"code_sha256_prefix":"13e901a00ee031f7","entry":"inspect_quantization","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/inference.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13e901a00ee031f7"}},{"code_sha256_prefix":"ac797b8a96bcf2ca","entry":"load_model","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/inference.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac797b8a96bcf2ca"}},{"code_sha256_prefix":"e18605c4e7b353e3","entry":"membership_fingerprint","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/datasets.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e18605c4e7b353e3"}},{"code_sha256_prefix":"5e5c34b5d00535fc","entry":"nvml_energy","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/telemetry.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/telemetry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5e5c34b5d00535fc"}},{"code_sha256_prefix":"85a97cdb29eadeb1","entry":"parse_answer","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/parsing.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/parsing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"85a97cdb29eadeb1"}},{"code_sha256_prefix":"513c4100cef1458f","entry":"parse_primary","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/parsing.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/parsing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"513c4100cef1458f"}},{"code_sha256_prefix":"34590ce57cb5ffcb","entry":"prompt_sha256","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/prompts.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34590ce57cb5ffcb"}},{"code_sha256_prefix":"fc31bf02f32733ef","entry":"recover_maintenance","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/parsing.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/parsing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc31bf02f32733ef"}},{"code_sha256_prefix":"e9274b5b09341e73","entry":"resolve_condition","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/conditions.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/conditions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e9274b5b09341e73"}},{"code_sha256_prefix":"d38645dd2388c7dc","entry":"select_manifest_records","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/subsets.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/subsets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d38645dd2388c7dc"}},{"code_sha256_prefix":"ce7f14457497f77f","entry":"summary","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/metrics.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce7f14457497f77f"}},{"code_sha256_prefix":"c0aa28da3b8ebdef","entry":"unit_id","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/datasets.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c0aa28da3b8ebdef"}},{"code_sha256_prefix":"e2708c53c57d1f0d","entry":"water_estimate","repo":"VectorInstitute/sustainable-rai-evaluation","repo_kind":"found_in_text","path":"src/evaluation_has_a_footprint/telemetry.py","file_url":"https://github.com/VectorInstitute/sustainable-rai-evaluation/blob/HEAD/src/evaluation_has_a_footprint/telemetry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2708c53c57d1f0d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_daily_20260901.json"},"syntology_extracted_results":null}