{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/assessing-language-model-deployment-with-risk","title":"Assessing Language Model Deployment with Risk Cards","arxiv_id":"2303.18190","date":"2023-03-31","proceeding":null,"authors":["Leon Derczynski","Hannah Rose Kirk","Vidhisha Balachandran","Sachin Kumar","Yulia Tsvetkov","M. R. Leiser","Saif Mohammad"],"abstract":"This paper introduces RiskCards, a framework for structured assessment and documentation of risks associated with an application of language models. As with all language, text generated by language models can be harmful, or used to bring about harm. Automating language generation adds both an element of scale and also more subtle or emergent undesirable tendencies to the generated text. Prior work establishes a wide variety of language model harms to many different actors: existing taxonomies identify categories of harms posed by language models; benchmarks establish automated tests of these harms; and documentation standards for models, tasks and datasets encourage transparent reporting. However, there is no risk-centric framework for documenting the complexity of a landscape in which some risks are shared across models and contexts, while others are specific, and where certain conditions may be required for risks to manifest as harms. RiskCards address this methodological gap by providing a generic framework for assessing the use of a given language model in a given scenario. Each RiskCard makes clear the routes for the risk to manifest harm, their placement in harm taxonomies, and example prompt-output pairs. While RiskCards are designed to be open-source, dynamic and participatory, we present a \"starter set\" of RiskCards taken from a broad literature survey, each of which details a concrete risk presentation. Language model RiskCards initiate a community knowledge base which permits the mapping of risks and harms to a specific model or its application scenario, ultimately contributing to a better, safer and shared understanding of the risk landscape.","url_abs":"https://arxiv.org/abs/2303.18190v1","url_pdf":"https://arxiv.org/pdf/2303.18190v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"assessing-language-model-deployment-with-risk","repo_url":"https://github.com/leondz/lm_risk_cards","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"assessing-language-model-deployment-with-risk","repo_url":"https://github.com/NVIDIA/garak","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"assessing-language-model-deployment-with-risk","repo_url":"https://github.com/leondz/garak","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.18190","atlas_url":"https://app.syntology.ai/?focus=2303.18190","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.18190"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leondz/garak","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leondz/lm_risk_cards","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NVIDIA/garak","reach":null}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"listed":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"05ad0e486ab38a94","entry":"get_parent_name","repo":"leondz/garak","repo_kind":"listed","path":"garak/cas.py","file_url":"https://github.com/leondz/garak/blob/HEAD/garak/cas.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"05ad0e486ab38a94"}},{"code_sha256_prefix":"45e2b3c580ddcce3","entry":"parse_spec_string","repo":"leondz/garak","repo_kind":"listed","path":"garak/_spec.py","file_url":"https://github.com/leondz/garak/blob/HEAD/garak/_spec.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"45e2b3c580ddcce3"}},{"code_sha256_prefix":"b78a0d9622df0be8","entry":"validate_intent_specifier","repo":"leondz/garak","repo_kind":"listed","path":"garak/_spec.py","file_url":"https://github.com/leondz/garak/blob/HEAD/garak/_spec.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b78a0d9622df0be8"}},{"code_sha256_prefix":"aed96232bc26f04c","entry":"parse_plugin_spec","repo":"leondz/garak","repo_kind":"listed","path":"garak/_config.py","file_url":"https://github.com/leondz/garak/blob/HEAD/garak/_config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"aed96232bc26f04c"}},{"code_sha256_prefix":"26d7bdd480185f15","entry":"parse_spec_file","repo":"leondz/garak","repo_kind":"listed","path":"garak/_spec.py","file_url":"https://github.com/leondz/garak/blob/HEAD/garak/_spec.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"26d7bdd480185f15"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}