{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-02280","title":"RACC: Representation-Aware Coverage Criteria for LLM Safety Testing","arxiv_id":"2602.02280","date":"2026-02-02","proceeding":null,"authors":["Zeming Wei","Zhixin Zhang","Chengcan Wu","Yihao Zhang","Xiaokun Luan","Meng Sun"],"abstract":"Large Language Models (LLMs) face severe safety risks from jailbreak attacks, yet current safety testing largely relies on static datasets and lacks systematic criteria to evaluate test suite quality and adequacy. While coverage criteria have proven effective for smaller neural networks, they are impractical for LLMs due to computational overhead and the entanglement of safety-critical signals with irrelevant neuron activations. To address these issues, we propose RACC (Representation-Aware Coverage Criteria), a set of coverage criteria specialized for LLM safety testing. RACC first extracts safety representations from the LLM's hidden states using a small calibration set of harmful prompts, then measures test prompts' concept activations against these directions, and finally computes coverage through six criteria assessing both individual and compositional safety concept coverage. Experiments on multiple LLMs and safety benchmarks show that RACC reliably rewards high-quality jailbreak test suites while remaining insensitive to redundant or invalid inputs, which is a key distinction that neuron-level criteria fail to make. We further demonstrate RACC's practical value in two applications, including test suite prioritization and attack prompt sampling, and validate its generalization across diverse settings and configurations. Overall, RACC provides a scalable and principled foundation for coverage-guided LLM safety testing.","url_abs":"https://arxiv.org/abs/2602.02280","url_pdf":"https://arxiv.org/pdf/2602.02280","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.02280","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.02280"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/weizeming/RACC","reach":null}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"d5c706a6ffcb1027","entry":"CBC","repo":"weizeming/RACC","repo_kind":"found_in_text","path":"compute_coverage.py","file_url":"https://github.com/weizeming/RACC/blob/HEAD/compute_coverage.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d5c706a6ffcb1027"}},{"code_sha256_prefix":"3bcf5a510a246720","entry":"FIC","repo":"weizeming/RACC","repo_kind":"found_in_text","path":"compute_coverage.py","file_url":"https://github.com/weizeming/RACC/blob/HEAD/compute_coverage.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3bcf5a510a246720"}},{"code_sha256_prefix":"d054656a92fb0ec9","entry":"PCAConceptExtractor","repo":"weizeming/RACC","repo_kind":"found_in_text","path":"compute_coverage.py","file_url":"https://github.com/weizeming/RACC/blob/HEAD/compute_coverage.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d054656a92fb0ec9"}},{"code_sha256_prefix":"51da3f3901ad5eac","entry":"PCC","repo":"weizeming/RACC","repo_kind":"found_in_text","path":"compute_coverage.py","file_url":"https://github.com/weizeming/RACC/blob/HEAD/compute_coverage.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"51da3f3901ad5eac"}},{"code_sha256_prefix":"c5b46dcef73836a7","entry":"SCC","repo":"weizeming/RACC","repo_kind":"found_in_text","path":"compute_coverage.py","file_url":"https://github.com/weizeming/RACC/blob/HEAD/compute_coverage.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5b46dcef73836a7"}},{"code_sha256_prefix":"1c6c7f08551a5aa0","entry":"SFC","repo":"weizeming/RACC","repo_kind":"found_in_text","path":"compute_coverage.py","file_url":"https://github.com/weizeming/RACC/blob/HEAD/compute_coverage.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c6c7f08551a5aa0"}},{"code_sha256_prefix":"a444d0791440c465","entry":"TKFC","repo":"weizeming/RACC","repo_kind":"found_in_text","path":"compute_coverage.py","file_url":"https://github.com/weizeming/RACC/blob/HEAD/compute_coverage.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a444d0791440c465"}},{"code_sha256_prefix":"cd4a7098b4a02b55","entry":"compute_racc","repo":"weizeming/RACC","repo_kind":"found_in_text","path":"compute_coverage.py","file_url":"https://github.com/weizeming/RACC/blob/HEAD/compute_coverage.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd4a7098b4a02b55"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}