{"url":"/method/relic","slug":"relic","name":"ReLIC","full_name":"ReLIC","full_name_withheld":false,"description_markdown":"**ReLIC**, or **Representation Learning via Invariant Causal Mechanisms**, is a self-supervised learning objective that enforces invariant prediction of proxy targets across augmentations through an invariance regularizer which yields improved generalization guarantees. \r\n\r\nWe can write the objective as:\r\n\r\n$$\r\n\\underset{X}{\\mathbb{E}} \\underset{\\sim\\_{l k}, a\\_{q \\mathcal{A}}}{\\mathbb{E}} \\sum_{b \\in\\left\\(a\\_{l k}, a\\_{q t}\\right\\)} \\mathcal{L}\\_{b}\\left(Y^{R}, f(X)\\right) \\text { s.t. } K L\\left(p^{d o\\left(a\\_{l k}\\right)}\\left(Y^{R} \\mid f(X)\\right), p^{d o\\left(a\\_{q t}\\right)}\\left(Y^{R} \\mid f(X)\\right)\\right) \\leq \\rho\r\n$$\r\n\r\nwhere $\\mathcal{L}$ is the proxy task loss and $K L$ is the Kullback-Leibler (KL) divergence. Note that any distance measure on distributions can be used in place of the KL divergence.\r\n\r\nConcretely, as proxy task we associate to every datapoint $x\\_{i}$ the label $y\\_{i}^{R}=i$. This corresponds to the instance discrimination task, commonly used in contrastive learning. We take pairs of points $\\left(x\\_{i}, x\\_{j}\\right)$ to compute similarity scores and use pairs of augmentations $a\\_{l k}=\\left(a\\_{l}, a\\_{k}\\right) \\in$ $\\mathcal{A} \\times \\mathcal{A}$ to perform a style intervention. Given a batch of samples $\\left\\(x\\_{i}\\right\\)\\_{i=1}^{N} \\sim \\mathcal{D}$, we use\r\n\r\n$$\r\np^{d o\\left(a\\_{l k}\\right)}\\left(Y^{R}=j \\mid f\\left(x\\_{i}\\right)\\right) \\propto \\exp \\left(\\phi\\left(f\\left(x\\_{i}^{a\\_{l}}\\right), h\\left(x\\_{j}^{a\\_{k}}\\right)\\right) / \\tau\\right)\r\n$$\r\n\r\nwith $x^{a}$ data augmented with $a$ and $\\tau$ a softmax temperature parameter. We encode $f$ using a neural network and choose $h$ to be related to $f$, e.g. $h=f$ or as a network with an exponential moving average of the weights of $f$ (e.g. target networks). To compare representations we use the function $\\phi\\left(f\\left(x\\_{i}\\right), h\\left(x\\_{j}\\right)\\right)=\\left\\langle g\\left(f\\left(x\\_{i}\\right)\\right), g\\left(h\\left(x\\_{j}\\right)\\right)\\right\\rangle$ where $g$ is a fully-connected neural network often called the critic.\r\n\r\nCombining these pieces, we learn representations by minimizing the following objective over the full set of data $x\\_{i} \\in \\mathcal{D}$ and augmentations $a_{l k} \\in \\mathcal{A} \\times \\mathcal{A}$\r\n\r\n$$\r\n-\\sum_{i=1}^{N} \\sum\\_{a\\_{l k}} \\log \\frac{\\exp \\left(\\phi\\left(f\\left(x\\_{i}^{a_{l}}\\right), h\\left(x\\_{i}^{a\\_{k}}\\right)\\right) / \\tau\\right)}{\\sum\\_{m=1}^{M} \\exp \\left(\\phi\\left(f\\left(x\\_{i}^{a\\_{l}}\\right), h\\left(x\\_{m}^{a\\_{k}}\\right)\\right) / \\tau\\right)}+\\alpha \\sum\\_{a\\_{l k}, a\\_{q t}} K L\\left(p^{d o\\left(a\\_{l k}\\right)}, p^{d o\\left(a\\_{q t}\\right)}\\right)\r\n$$\r\n\r\nwith $M$ the number of points we use to construct the contrast set and $\\alpha$ the weighting of the invariance penalty. The shorthand $p^{d o(a)}$ is used for $p^{d o(a)}\\left(Y^{R}=j \\mid f\\left(x\\_{i}\\right)\\right)$. The Figure shows a schematic of the RELIC objective.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Representation Learning via Invariant Causal Mechanisms","paper":"/paper/representation-learning-via-invariant-causal-1","first_author":"Jovana Mitrovic","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/representation-learning-via-invariant-causal-1"},"source":{"url":"https://arxiv.org/abs/2010.07922v1","title":"Representation Learning via Invariant Causal Mechanisms","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Self-Supervised Learning","url":"/methods/category/self-supervised-learning","pwc_aliases":[]}],"n_papers_tagged":12,"archive_num_papers":12,"papers_newest_first":[{"paper":null,"title":"RELIC: Evaluating Compositional Instruction Following via Language Recognition","date":"2025-06-05","arxiv_id":"2506.05205","n_code_links":0,"syntology":null},{"paper":null,"title":"A Collaborative Jade Recognition System for Mobile Devices Based on Lightweight and Large Models","date":"2025-02-20","arxiv_id":"2502.14332","n_code_links":0,"syntology":null},{"paper":null,"title":"An RNA condensate model for the origin of life","date":"2024-12-06","arxiv_id":"2412.05396","n_code_links":0,"syntology":null},{"paper":null,"title":"MRIFE: A Mask-Recovering and Interactive-Feature-Enhancing Semantic Segmentation Network For Relic Landslide Detection","date":"2024-11-26","arxiv_id":"2411.17167","n_code_links":0,"syntology":null},{"paper":null,"title":"Machine-Learning Analysis of Radiative Decays to Dark Matter at the LHC","date":"2024-10-17","arxiv_id":"2410.13799","n_code_links":0,"syntology":null},{"paper":"/paper/relic-a-recipe-for-64k-steps-of-in-context","title":"ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI","date":"2024-10-03","arxiv_id":"2410.02751","n_code_links":1,"syntology":{"ran":3,"of":7,"unverified":4,"pointer_only":7}},{"paper":null,"title":"Symbolic Regression for Beyond the Standard Model Physics","date":"2024-05-28","arxiv_id":"2405.18471","n_code_links":0,"syntology":null},{"paper":"/paper/self-supervised-learning-through-efference","title":"Self-Supervised Learning Through Efference Copies","date":"2022-10-17","arxiv_id":"2210.09224","n_code_links":1,"syntology":null},{"paper":null,"title":"Mixed Anhydrides at the Intersection Between Peptide and RNA Autocatalytic Sets: Evolution of Biological Coding","date":"2022-05-25","arxiv_id":"2208.01491","n_code_links":0,"syntology":null},{"paper":"/paper/pushing-the-limits-of-self-supervised-resnets","title":"Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?","date":"2022-01-13","arxiv_id":"2201.05119","n_code_links":1,"syntology":{"ran":0,"of":14,"unverified":14,"pointer_only":0}},{"paper":"/paper/coberl-contrastive-bert-for-reinforcement","title":"CoBERL: Contrastive BERT for Reinforcement Learning","date":"2021-07-12","arxiv_id":"2107.05431","n_code_links":2,"syntology":{"ran":3,"of":3,"unverified":0,"pointer_only":0}},{"paper":"/paper/representation-learning-via-invariant-causal-1","title":"Representation Learning via Invariant Causal Mechanisms","date":"2020-10-15","arxiv_id":"2010.07922","n_code_links":2,"syntology":{"ran":2,"of":2,"unverified":0,"pointer_only":1}}],"papers_shown":12,"tasks":[{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":3},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":2},{"task":"/task/image-classification","name":"Image Classification","papers":2},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":2},{"task":"/task/representation-learning","name":"Representation Learning","papers":2},{"task":"/task/self-supervised-image-classification","name":"Self-Supervised Image Classification","papers":2},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":2},{"task":"/task/image-classification","name":"image-classification","papers":2},{"task":"/task/few-shot-imitation-learning","name":"Few-Shot Imitation Learning","papers":1},{"task":"/task/imitation-learning","name":"Imitation Learning","papers":1},{"task":"/task/in-context-learning","name":"In-Context Learning","papers":1},{"task":null,"name":"In-Context Reinforcement Learning","papers":1},{"task":"/task/instruction-following","name":"Instruction Following","papers":1},{"task":"/task/linear-evaluation","name":"Linear evaluation","papers":1},{"task":"/task/object-detection","name":"Object Detection","papers":1},{"task":"/task/out-of-distribution-generalization","name":"Out-of-Distribution Generalization","papers":1},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":1},{"task":"/task/semi-supervised-image-classification","name":"Semi-Supervised Image Classification","papers":1},{"task":"/task/symbolic-regression","name":"Symbolic Regression","papers":1},{"task":"/task/model","name":"model","papers":1}],"tasks_shown":20,"n_tasks":23,"usage_by_year":[{"year":"2020","papers":1},{"year":"2021","papers":1},{"year":"2022","papers":3},{"year":"2024","papers":5},{"year":"2025","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/relic"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}