{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-10913","title":"LLM2Vec-Gen: Generative Embeddings from Large Language Models","arxiv_id":"2603.10913","date":"2026-03-11","proceeding":null,"authors":["Parishad BehnamGhader","Vaibhav Adlakha","Fabian David Schmidt","Nicolas Chapados","Marius Mosbach","Siva Reddy"],"abstract":"Fine-tuning LLM-based text embedders via contrastive learning maps inputs and outputs into a new representational space, discarding the LLM's output semantics. We propose LLM2Vec-Gen, a self-supervised alternative that instead produces embeddings directly in the LLM's output space by learning to represent the model's potential response. Specifically, trainable special tokens are appended to the input and optimized to compress the LLM's own response into a fixed-length embedding, guided by an unsupervised embedding teacher and a reconstruction objective. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 8.8% over the unsupervised embedding teacher. Since the embeddings preserve the LLM's response-space semantics, they inherit capabilities such as safety alignment (up to 22.6% reduction in harmful content retrieval) and reasoning (up to 35.6% improvement on reasoning-intensive retrieval). Finally, the learned embeddings are also interpretable: they can be decoded back into text to reveal their semantic content. Our code, data, and pretrained models are publicly available at https://github.com/McGill-NLP/llm2vec-gen.","url_abs":"https://arxiv.org/abs/2603.10913","url_pdf":"https://arxiv.org/pdf/2603.10913","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.10913","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.10913"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/McGill-NLP/llm2vec-gen","reach":null}],"summary":{"ran":1,"ran_draft_wrong":2,"ran_fixture":1,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f615629a46cf0ccd","entry":"ProjectionModel","repo":"McGill-NLP/llm2vec-gen","repo_kind":"found_in_text","path":"llm2vec_gen/models/modeling_encoder_decoder.py","file_url":"https://github.com/McGill-NLP/llm2vec-gen/blob/HEAD/llm2vec_gen/models/modeling_encoder_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f615629a46cf0ccd"}},{"code_sha256_prefix":"58f0830a33792738","entry":"custom_constructor","repo":"McGill-NLP/llm2vec-gen","repo_kind":"found_in_text","path":"llm2vec_gen/models/modeling_encoder_decoder.py","file_url":"https://github.com/McGill-NLP/llm2vec-gen/blob/HEAD/llm2vec_gen/models/modeling_encoder_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58f0830a33792738"}},{"code_sha256_prefix":"5ba2d9c2ed0f59ff","entry":"prepend_prompt_to_left_padded_tok_embeds","repo":"McGill-NLP/llm2vec-gen","repo_kind":"found_in_text","path":"llm2vec_gen/models/modeling_encoder_decoder.py","file_url":"https://github.com/McGill-NLP/llm2vec-gen/blob/HEAD/llm2vec_gen/models/modeling_encoder_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ba2d9c2ed0f59ff"}},{"code_sha256_prefix":"2d28f2510ec94d25","entry":"safe_load_config","repo":"McGill-NLP/llm2vec-gen","repo_kind":"found_in_text","path":"llm2vec_gen/models/modeling_encoder_decoder.py","file_url":"https://github.com/McGill-NLP/llm2vec-gen/blob/HEAD/llm2vec_gen/models/modeling_encoder_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d28f2510ec94d25"}},{"code_sha256_prefix":"4f5fca7e6d82b118","entry":"EncoderDecoderModel","repo":"McGill-NLP/llm2vec-gen","repo_kind":"found_in_text","path":"llm2vec_gen/models/modeling_encoder_decoder.py","file_url":"https://github.com/McGill-NLP/llm2vec-gen/blob/HEAD/llm2vec_gen/models/modeling_encoder_decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f5fca7e6d82b118"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}