{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/inverse-constitutional-ai-compressing","title":"Inverse Constitutional AI: Compressing Preferences into Principles","arxiv_id":"2406.06560","date":"2024-06-02","proceeding":null,"authors":["Arduin Findeis","Timo Kaufmann","Eyke Hüllermeier","Samuel Albanie","Robert Mullins"],"abstract":"Feedback data is widely used for fine-tuning and evaluating state-of-the-art AI models. Pairwise text preferences, where human or AI annotators select the \"better\" of two options, are particularly common. Such preferences are used to train (reward) models or to rank models with aggregate statistics. For many applications it is desirable to understand annotator preferences in addition to modelling them - not least because extensive prior work has shown various unintended biases in preference datasets. Yet, preference datasets remain challenging to interpret. Neither black-box reward models nor statistics can answer why one text is preferred over another. Manual interpretation of the numerous (long) response pairs is usually equally infeasible. In this paper, we introduce the Inverse Constitutional AI (ICAI) problem, formulating the interpretation of pairwise text preference data as a compression task. In constitutional AI, a set of principles (a constitution) is used to provide feedback and fine-tune AI models. ICAI inverts this process: given a feedback dataset, we aim to extract a constitution that best enables a large language model (LLM) to reconstruct the original annotations. We propose a corresponding ICAI algorithm and validate its generated constitutions quantitatively based on annotation reconstruction accuracy on several datasets: (a) synthetic feedback data with known principles; (b) AlpacaEval cross-annotated human feedback data; (c) crowdsourced Chatbot Arena data; and (d) PRISM data from diverse demographic groups. As a short and interpretable representation of the original dataset, generated constitutions have many potential use cases: help identify undesirable annotator biases, understand model performance better, scale feedback to unseen data, or adapt models to individual user or group preferences. We release the source code at https://github.com/rdnfn/icai.","url_abs":"https://arxiv.org/abs/2406.06560v2","url_pdf":"https://arxiv.org/pdf/2406.06560v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"inverse-constitutional-ai-compressing","repo_url":"https://github.com/rdnfn/icai","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"chatbot","task_name":"Chatbot"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.06560","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.06560"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/rdnfn/icai","reach":null}],"summary":{"ran":3,"ran_draft_wrong":3,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fad9c323896389d9","entry":"AlpacaEvalAnnotatorConfig","repo":"rdnfn/icai","repo_kind":"official","path":"src/inverse_cai/algorithm/proposal.py","file_url":"https://github.com/rdnfn/icai/blob/HEAD/src/inverse_cai/algorithm/proposal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fad9c323896389d9"}},{"code_sha256_prefix":"3c4ba0e3b3c33583","entry":"AnnotatorConfig","repo":"rdnfn/icai","repo_kind":"official","path":"src/inverse_cai/algorithm/proposal.py","file_url":"https://github.com/rdnfn/icai/blob/HEAD/src/inverse_cai/algorithm/proposal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c4ba0e3b3c33583"}},{"code_sha256_prefix":"59a4167a57113c68","entry":"FunctionAnnotatorConfig","repo":"rdnfn/icai","repo_kind":"official","path":"src/inverse_cai/algorithm/proposal.py","file_url":"https://github.com/rdnfn/icai/blob/HEAD/src/inverse_cai/algorithm/proposal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"59a4167a57113c68"}},{"code_sha256_prefix":"cf0ddec0199e9097","entry":"clean_principle_str","repo":"rdnfn/icai","repo_kind":"official","path":"src/inverse_cai/algorithm/proposal.py","file_url":"https://github.com/rdnfn/icai/blob/HEAD/src/inverse_cai/algorithm/proposal.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf0ddec0199e9097"}},{"code_sha256_prefix":"56041d176337cc61","entry":"get_preferred_text","repo":"rdnfn/icai","repo_kind":"official","path":"src/inverse_cai/algorithm/proposal.py","file_url":"https://github.com/rdnfn/icai/blob/HEAD/src/inverse_cai/algorithm/proposal.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56041d176337cc61"}},{"code_sha256_prefix":"d1252752dc993a17","entry":"get_rejected_text","repo":"rdnfn/icai","repo_kind":"official","path":"src/inverse_cai/algorithm/proposal.py","file_url":"https://github.com/rdnfn/icai/blob/HEAD/src/inverse_cai/algorithm/proposal.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d1252752dc993a17"}},{"code_sha256_prefix":"2417283750496843","entry":"ExpConfig","repo":"rdnfn/icai","repo_kind":"official","path":"src/inverse_cai/algorithm/proposal.py","file_url":"https://github.com/rdnfn/icai/blob/HEAD/src/inverse_cai/algorithm/proposal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2417283750496843"}},{"code_sha256_prefix":"d93fe6772a75b268","entry":"PromptConfig","repo":"rdnfn/icai","repo_kind":"official","path":"src/inverse_cai/algorithm/proposal.py","file_url":"https://github.com/rdnfn/icai/blob/HEAD/src/inverse_cai/algorithm/proposal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d93fe6772a75b268"}},{"code_sha256_prefix":"7aeeac243f7d0d42","entry":"generate_principles_from_feedback","repo":"rdnfn/icai","repo_kind":"official","path":"src/inverse_cai/algorithm/proposal.py","file_url":"https://github.com/rdnfn/icai/blob/HEAD/src/inverse_cai/algorithm/proposal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7aeeac243f7d0d42"}},{"code_sha256_prefix":"7e33551b08a8b93b","entry":"generate_principles_from_single_ranking","repo":"rdnfn/icai","repo_kind":"official","path":"src/inverse_cai/algorithm/proposal.py","file_url":"https://github.com/rdnfn/icai/blob/HEAD/src/inverse_cai/algorithm/proposal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e33551b08a8b93b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}