{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/personalized-representation-from-personalized","title":"Personalized Representation from Personalized Generation","arxiv_id":"2412.16156","date":"2024-12-20","proceeding":null,"authors":["Shobhita Sundaram","Julia Chae","Yonglong Tian","Sara Beery","Phillip Isola"],"abstract":"Modern vision models excel at general purpose downstream tasks. It is unclear, however, how they may be used for personalized vision tasks, which are both fine-grained and data-scarce. Recent works have successfully applied synthetic data to general-purpose representation learning, while advances in T2I diffusion models have enabled the generation of personalized images from just a few real examples. Here, we explore a potential connection between these ideas, and formalize the challenge of using personalized synthetic data to learn personalized representations, which encode knowledge about an object of interest and may be flexibly applied to any downstream task relating to the target object. We introduce an evaluation suite for this challenge, including reformulations of two existing datasets and a novel dataset explicitly constructed for this purpose, and propose a contrastive learning approach that makes creative use of image generators. We show that our method improves personalized representation learning for diverse downstream tasks, from recognition to segmentation, and analyze characteristics of image generation approaches that are key to this gain.","url_abs":"https://arxiv.org/abs/2412.16156v1","url_pdf":"https://arxiv.org/pdf/2412.16156v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"personalized-representation-from-personalized","repo_url":"https://github.com/ssundaram21/personalized-rep","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.16156","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.16156"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ssundaram21/personalized-rep","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":14},"by_repo_kind":{"official":{"samples":15,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"55120f2026b56aa2","entry":"drop_path","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"models/vision_transformer.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/models/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55120f2026b56aa2"}},{"code_sha256_prefix":"9dd2cde305dd9e67","entry":"classification","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"downstream/classification.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/downstream/classification.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9dd2cde305dd9e67"}},{"code_sha256_prefix":"52a4e7b1ef28e2dd","entry":"compute_auc","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"downstream/classification.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/downstream/classification.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52a4e7b1ef28e2dd"}},{"code_sha256_prefix":"36a2e6971921cd26","entry":"get_loss_fn","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"contrastive/losses.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/contrastive/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36a2e6971921cd26"}},{"code_sha256_prefix":"7776840ed560eb2e","entry":"get_negative","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"contrastive/make_triplets.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/contrastive/make_triplets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7776840ed560eb2e"}},{"code_sha256_prefix":"0dbfcb6e6094bdf8","entry":"get_positives","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"contrastive/make_triplets.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/contrastive/make_triplets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0dbfcb6e6094bdf8"}},{"code_sha256_prefix":"5a2742d214296afa","entry":"get_preprocess_shape","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"downstream/dense_tasks.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/downstream/dense_tasks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a2742d214296afa"}},{"code_sha256_prefix":"53d3d9f0c6d4be38","entry":"load_clip_as_dino","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"models/load_clip_as_dino.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/models/load_clip_as_dino.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53d3d9f0c6d4be38"}},{"code_sha256_prefix":"d3cfc3dc84296355","entry":"load_mae_as_vit","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"models/load_mae_as_vit.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/models/load_mae_as_vit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3cfc3dc84296355"}},{"code_sha256_prefix":"56f43e8386219d7a","entry":"pil_loader","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"dataset/image_dataset.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/dataset/image_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56f43e8386219d7a"}},{"code_sha256_prefix":"ff4a6945ab54d2a4","entry":"prepare_mask","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"downstream/dense_tasks.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/downstream/dense_tasks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff4a6945ab54d2a4"}},{"code_sha256_prefix":"cdcc264ab355618c","entry":"preprocess","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"downstream/dense_tasks.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/downstream/dense_tasks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cdcc264ab355618c"}},{"code_sha256_prefix":"8e2ab79c67b1453f","entry":"sample_triplet","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"contrastive/make_triplets.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/contrastive/make_triplets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e2ab79c67b1453f"}},{"code_sha256_prefix":"c9f9ff84dc5ae644","entry":"vit_small","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"models/vision_transformer.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/models/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9f9ff84dc5ae644"}},{"code_sha256_prefix":"256e2853fd88f06e","entry":"vit_tiny","repo":"ssundaram21/personalized-rep","repo_kind":"official","path":"models/vision_transformer.py","file_url":"https://github.com/ssundaram21/personalized-rep/blob/HEAD/models/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"256e2853fd88f06e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}