{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-12324","title":"When AI Is Your Pastor: A Benchmark for Theological Triage and Pastoral Guidance in Large Language Models","arxiv_id":"2608.12324","date":"2026-05-29","proceeding":null,"authors":["Alex Chao"],"abstract":"People increasingly ask large language models (LLMs) for counsel on questions of faith, doctrine, and pastoral care. These questions are not ordinary information requests. Some ask about core Christian beliefs, some ask about real disagreements among faithful traditions, some require humility because the issue is prudential, and some are pastoral situations where safety and human referral matter more than theological completeness. Existing benchmarks do not evaluate this structure. We introduce FMG-Bench, the Faith & Moral Guidance Benchmark, a 120-scenario benchmark for evaluating large language model behavior in English-language Christian theological triage and pastoral guidance contexts. FMG-Bench v1 evaluates 14 advanced models across 8,792 scored responses, comparing raw model behavior with three guided instruction settings. In our production run, placing models inside a structured harness improves over raw model behavior by +3.96 points on average, with every model improving. The most safety-critical finding is a +10.8 point gain in escalation appropriateness -- whether AI systems recognize when pastoral, clinical, legal, or emergency support is needed. The guided settings also improve robustness, meaning consistency when questions are reworded or pressured (92.88 to 98.02 stability). Asking a model to compare perspectives helps in secondary-doctrine questions but can be counterproductive when applied to primary doctrine or urgent pastoral situations. The benchmark is a measurement tool, not an endorsement of AI systems as pastoral authorities.","url_abs":"https://arxiv.org/abs/2608.12324","url_pdf":"https://arxiv.org/pdf/2608.12324","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2608.12324","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.12324"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/FideAI/fmg-bench","reach":null}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"03fa718561a87743","entry":"load_scenario_set_manifest","repo":"FideAI/fmg-bench","repo_kind":"found_in_text","path":"benchmark/system_benchmark/publication.py","file_url":"https://github.com/FideAI/fmg-bench/blob/HEAD/benchmark/system_benchmark/publication.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"03fa718561a87743"}},{"code_sha256_prefix":"424f79ce463b54a7","entry":"model_slug","repo":"FideAI/fmg-bench","repo_kind":"found_in_text","path":"benchmark/system_benchmark/models.py","file_url":"https://github.com/FideAI/fmg-bench/blob/HEAD/benchmark/system_benchmark/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"424f79ce463b54a7"}},{"code_sha256_prefix":"5ad3b5e0903bbf77","entry":"normalize_call_result","repo":"FideAI/fmg-bench","repo_kind":"found_in_text","path":"benchmark/system_benchmark/models.py","file_url":"https://github.com/FideAI/fmg-bench/blob/HEAD/benchmark/system_benchmark/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5ad3b5e0903bbf77"}},{"code_sha256_prefix":"c4c148ae1ac366a3","entry":"parse_judge_models","repo":"FideAI/fmg-bench","repo_kind":"found_in_text","path":"benchmark/run_fmg_bench.py","file_url":"https://github.com/FideAI/fmg-bench/blob/HEAD/benchmark/run_fmg_bench.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c4c148ae1ac366a3"}},{"code_sha256_prefix":"f2162f379c90207b","entry":"import_human_review_csv","repo":"FideAI/fmg-bench","repo_kind":"found_in_text","path":"benchmark/system_benchmark/calibration.py","file_url":"https://github.com/FideAI/fmg-bench/blob/HEAD/benchmark/system_benchmark/calibration.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f2162f379c90207b"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}