{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-calibration-of-llm-based-guard-models-for","title":"On Calibration of LLM-based Guard Models for Reliable Content Moderation","arxiv_id":"2410.10414","date":"2024-10-14","proceeding":null,"authors":["Hongfu Liu","Hengguan Huang","Xiangming Gu","Hao Wang","Ye Wang"],"abstract":"Large language models (LLMs) pose significant risks due to the potential for generating harmful content or users attempting to evade guardrails. Existing studies have developed LLM-based guard models designed to moderate the input and output of threat LLMs, ensuring adherence to safety policies by blocking content that violates these protocols upon deployment. However, limited attention has been given to the reliability and calibration of such guard models. In this work, we empirically conduct comprehensive investigations of confidence calibration for 9 existing LLM-based guard models on 12 benchmarks in both user input and model output classification. Our findings reveal that current LLM-based guard models tend to 1) produce overconfident predictions, 2) exhibit significant miscalibration when subjected to jailbreak attacks, and 3) demonstrate limited robustness to the outputs generated by different types of response models. Additionally, we assess the effectiveness of post-hoc calibration methods to mitigate miscalibration. We demonstrate the efficacy of temperature scaling and, for the first time, highlight the benefits of contextual calibration for confidence calibration of guard models, particularly in the absence of validation sets. Our analysis and experiments underscore the limitations of current LLM-based guard models and provide valuable insights for the future development of well-calibrated guard models toward more reliable content moderation. We also advocate for incorporating reliability evaluation of confidence calibration when releasing future LLM-based guard models.","url_abs":"https://arxiv.org/abs/2410.10414v2","url_pdf":"https://arxiv.org/pdf/2410.10414v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-calibration-of-llm-based-guard-models-for","repo_url":"https://github.com/waffle-liu/calibration_guard_model","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"blocking","task_name":"Blocking"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.10414","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.10414"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/waffle-liu/calibration_guard_model","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4c39995866b47811","entry":"calibrate_py","repo":"waffle-liu/calibration_guard_model","repo_kind":"official","path":"eval_model_dep.py","file_url":"https://github.com/waffle-liu/calibration_guard_model/blob/HEAD/eval_model_dep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c39995866b47811"}},{"code_sha256_prefix":"7dc9736cdb2847c4","entry":"get_equal_bins","repo":"waffle-liu/calibration_guard_model","repo_kind":"official","path":"calibration/utils.py","file_url":"https://github.com/waffle-liu/calibration_guard_model/blob/HEAD/calibration/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7dc9736cdb2847c4"}},{"code_sha256_prefix":"d1ebc0d7093a4203","entry":"get_equal_prob_bins","repo":"waffle-liu/calibration_guard_model","repo_kind":"official","path":"calibration/utils.py","file_url":"https://github.com/waffle-liu/calibration_guard_model/blob/HEAD/calibration/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1ebc0d7093a4203"}},{"code_sha256_prefix":"8e9dfd4ba4bf9fa3","entry":"split","repo":"waffle-liu/calibration_guard_model","repo_kind":"official","path":"calibration/utils.py","file_url":"https://github.com/waffle-liu/calibration_guard_model/blob/HEAD/calibration/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e9dfd4ba4bf9fa3"}},{"code_sha256_prefix":"8028fee51e12d3b0","entry":"compute_results_classifier_hug","repo":"waffle-liu/calibration_guard_model","repo_kind":"official","path":"eval_utils.py","file_url":"https://github.com/waffle-liu/calibration_guard_model/blob/HEAD/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8028fee51e12d3b0"}},{"code_sha256_prefix":"9a447532c1113a53","entry":"moderation_prompt_for_chat","repo":"waffle-liu/calibration_guard_model","repo_kind":"official","path":"eval_utils.py","file_url":"https://github.com/waffle-liu/calibration_guard_model/blob/HEAD/eval_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a447532c1113a53"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}