{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/robust-concept-erasure-via-kernelized-rate-1","title":"Robust Concept Erasure via Kernelized Rate-Distortion Maximization","arxiv_id":"2312.00194","date":"2023-11-30","proceeding":"NeurIPS 2023 11","authors":["Somnath Basu Roy Chowdhury","Nicholas Monath","Avinava Dubey","Amr Ahmed","Snigdha Chaturvedi"],"abstract":"Distributed representations provide a vector space that captures meaningful relationships between data instances. The distributed nature of these representations, however, entangles together multiple attributes or concepts of data instances (e.g., the topic or sentiment of a text, characteristics of the author (age, gender, etc), etc). Recent work has proposed the task of concept erasure, in which rather than making a concept predictable, the goal is to remove an attribute from distributed representations while retaining other information from the original representation space as much as possible. In this paper, we propose a new distance metric learning-based objective, the Kernelized Rate-Distortion Maximizer (KRaM), for performing concept erasure. KRaM fits a transformation of representations to match a specified distance measure (defined by a labeled concept to erase) using a modified rate-distortion function. Specifically, KRaM's objective function aims to make instances with similar concept labels dissimilar in the learned representation space while retaining other information. We find that optimizing KRaM effectively erases various types of concepts: categorical, continuous, and vector-valued variables from data representations across diverse domains. We also provide a theoretical analysis of several properties of KRaM's objective. To assess the quality of the learned representations, we propose an alignment score to evaluate their similarity with the original representation space. Additionally, we conduct experiments to showcase KRaM's efficacy in various settings, from erasing binary gender variables in word embeddings to vector-valued variables in GPT-3 representations.","url_abs":"https://arxiv.org/abs/2312.00194v1","url_pdf":"https://arxiv.org/pdf/2312.00194v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"robust-concept-erasure-via-kernelized-rate-1","repo_url":"https://github.com/brcsomnath/kram","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"metric-learning","task_name":"Metric Learning"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2312.00194","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.00194"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/brcsomnath/KRaM","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brcsomnath/kram","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":18,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":21,"ran":20,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"49a93e98ae41a86f","entry":"RateDistortion","repo":"brcsomnath/kram","repo_kind":"official","path":"src/loss.py","file_url":"https://github.com/brcsomnath/kram/blob/HEAD/src/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49a93e98ae41a86f"}},{"code_sha256_prefix":"efd704682a41f0a3","entry":"RateDistortionContinuous","repo":"brcsomnath/kram","repo_kind":"official","path":"src/loss.py","file_url":"https://github.com/brcsomnath/kram/blob/HEAD/src/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efd704682a41f0a3"}},{"code_sha256_prefix":"8e8e1bf9b627b971","entry":"cauchy_kernel","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/metrics.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e8e1bf9b627b971"}},{"code_sha256_prefix":"590b4d2c85eae0a0","entry":"classify","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"590b4d2c85eae0a0"}},{"code_sha256_prefix":"8a14ef4db6394aa7","entry":"correlation","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/evaluate.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a14ef4db6394aa7"}},{"code_sha256_prefix":"8913e4e11f92191f","entry":"debias_by_specific_directions","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/baselines/inlp.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/baselines/inlp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8913e4e11f92191f"}},{"code_sha256_prefix":"b5e6d7d6e3f5eb54","entry":"evaluate_regression","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5e6d7d6e3f5eb54"}},{"code_sha256_prefix":"c4b2effb5d8110a1","entry":"gaussian_kernel","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/metrics.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4b2effb5d8110a1"}},{"code_sha256_prefix":"b26df1b3be7a084e","entry":"get_projection_to_intersection_of_nullspaces","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/baselines/inlp.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/baselines/inlp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b26df1b3be7a084e"}},{"code_sha256_prefix":"3f72ad49af8d0e6b","entry":"get_representations_from_loader","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/baselines/farm.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/baselines/farm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f72ad49af8d0e6b"}},{"code_sha256_prefix":"4484e72f4b964121","entry":"get_representations_from_loader","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/baselines/mi_debiasing.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/baselines/mi_debiasing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4484e72f4b964121"}},{"code_sha256_prefix":"f75e44f7fc419025","entry":"get_rowspace_projection","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/baselines/inlp.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/baselines/inlp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f75e44f7fc419025"}},{"code_sha256_prefix":"0ff8884ceed2cb83","entry":"knn","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/evaluate.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ff8884ceed2cb83"}},{"code_sha256_prefix":"abef5bdd6d4d119a","entry":"label_to_membership","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/loss.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/loss.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"abef5bdd6d4d119a"}},{"code_sha256_prefix":"4b0b366504cec402","entry":"laplace_kernel","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/metrics.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b0b366504cec402"}},{"code_sha256_prefix":"69fd31f5a04ad9e2","entry":"load_data","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/data_load.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/data_load.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69fd31f5a04ad9e2"}},{"code_sha256_prefix":"55c2f56cc152566e","entry":"load_deepmoji","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/data_load.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/data_load.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55c2f56cc152566e"}},{"code_sha256_prefix":"62dfecd92146a155","entry":"one_hot","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/loss.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/loss.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62dfecd92146a155"}},{"code_sha256_prefix":"c27d027c708f2a99","entry":"pairwise_dist","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/baselines/farm.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/baselines/farm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c27d027c708f2a99"}},{"code_sha256_prefix":"e6ca500737cb0b1c","entry":"regression_predict","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6ca500737cb0b1c"}},{"code_sha256_prefix":"d127dafd8f5d0f8a","entry":"read_dataset","repo":"brcsomnath/KRaM","repo_kind":"official","path":"src/data_load.py","file_url":"https://github.com/brcsomnath/KRaM/blob/HEAD/src/data_load.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d127dafd8f5d0f8a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}