{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-a-bayesian-oracle-prevent-harm-from-an","title":"Can a Bayesian Oracle Prevent Harm from an Agent?","arxiv_id":"2408.05284","date":"2024-08-09","proceeding":null,"authors":["Yoshua Bengio","Michael K. Cohen","Nikolay Malkin","Matt MacDermott","Damiano Fornasiere","Pietro Greiner","Younesse Kaddar"],"abstract":"Is there a way to design powerful AI systems based on machine learning methods that would satisfy probabilistic safety guarantees? With the long-term goal of obtaining a probabilistic guarantee that would apply in every context, we consider estimating a context-dependent bound on the probability of violating a given safety specification. Such a risk evaluation would need to be performed at run-time to provide a guardrail against dangerous actions of an AI. Noting that different plausible hypotheses about the world could produce very different outcomes, and because we do not know which one is right, we derive bounds on the safety violation probability predicted under the true but unknown hypothesis. Such bounds could be used to reject potentially dangerous actions. Our main results involve searching for cautious but plausible hypotheses, obtained by a maximization that involves Bayesian posteriors over hypotheses. We consider two forms of this result, in the iid case and in the non-iid case, and conclude with open problems towards turning such theoretical results into practical AI guardrails.","url_abs":"https://arxiv.org/abs/2408.05284v2","url_pdf":"https://arxiv.org/pdf/2408.05284v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-a-bayesian-oracle-prevent-harm-from-an","repo_url":"https://github.com/saifh-github/conservative-bayesian-public","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2408.05284","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.05284"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/saifh-github/conservative-bayesian-public","reach":{"status":"ok"}}],"summary":{"ran":9,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"652ef07912a41d35","entry":"custom_metric","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"run_new_non_iid.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/run_new_non_iid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"652ef07912a41d35"}},{"code_sha256_prefix":"6f24a98afccf37ed","entry":"format_to_1sf","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"utils/plotting.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/utils/plotting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f24a98afccf37ed"}},{"code_sha256_prefix":"6b778d16f3f9e142","entry":"get_device","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b778d16f3f9e142"}},{"code_sha256_prefix":"08cf7181c4c2826b","entry":"get_latest","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"make_non_iid_plots.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/make_non_iid_plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"08cf7181c4c2826b"}},{"code_sha256_prefix":"bda67224d4d53202","entry":"get_latest","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"make_tightness_plots.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/make_tightness_plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bda67224d4d53202"}},{"code_sha256_prefix":"99b0dff2bb68bf7e","entry":"get_latest","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"make_transposed_non_iid_plots.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/make_transposed_non_iid_plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99b0dff2bb68bf7e"}},{"code_sha256_prefix":"9e130cd09b1406c0","entry":"get_mean_and_error","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e130cd09b1406c0"}},{"code_sha256_prefix":"bab04bc468514d50","entry":"log_survival_function","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"utils/utils_numerical.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/utils/utils_numerical.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bab04bc468514d50"}},{"code_sha256_prefix":"49966bbac52a3d3f","entry":"string_repr_current_hyperparams","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"run_new_non_iid.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/run_new_non_iid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"49966bbac52a3d3f"}},{"code_sha256_prefix":"236b4e6d0cd05076","entry":"fig_deaths_reward_custom_metric_vs_alpha_at_threshold","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"utils/plotting.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/utils/plotting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"236b4e6d0cd05076"}},{"code_sha256_prefix":"030df2904b846a32","entry":"plot_deaths_and_reward_vs_alpha","repo":"saifh-github/conservative-bayesian-public","repo_kind":"official","path":"utils/plotting.py","file_url":"https://github.com/saifh-github/conservative-bayesian-public/blob/HEAD/utils/plotting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"030df2904b846a32"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}