{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/following-the-autoregressive-nature-of-llm","title":"Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment","arxiv_id":"2502.11401","date":"2025-02-17","proceeding":null,"authors":["Jingcheng Deng","Zhongtao Jiang","Liang Pang","Liwei Chen","Kun Xu","Zihao Wei","HuaWei Shen","Xueqi Cheng"],"abstract":"A new trend uses LLMs as dense text encoders via contrastive learning. However, since LLM embeddings predict the probability distribution of the next token, they are inherently generative and distributive, conflicting with contrastive learning, which requires embeddings to capture full-text semantics and align via cosine similarity. This discrepancy hinders the full utilization of LLMs' pre-training capabilities, resulting in inefficient learning. In response to this issue, we propose AutoRegEmbed, a new contrastive learning method built on embedding conditional probability distributions, which integrates two core tasks: information compression and conditional distribution alignment. The information compression task encodes text into the embedding space, ensuring that the embedding vectors capture global semantics. The conditional distribution alignment task focuses on aligning text embeddings with positive samples embeddings by leveraging the conditional distribution of embeddings while simultaneously reducing the likelihood of generating negative samples from text embeddings, thereby achieving embedding alignment and uniformity. Experimental results demonstrate that our method significantly outperforms traditional contrastive learning approaches and achieves performance comparable to state-of-the-art models when using the same amount of data.","url_abs":"https://arxiv.org/abs/2502.11401v1","url_pdf":"https://arxiv.org/pdf/2502.11401v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"following-the-autoregressive-nature-of-llm","repo_url":"https://github.com/trustedllm/autoregembed","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.11401","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.11401"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/trustedllm/autoregembed","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/TrustedLLM/AutoRegEmbed","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"953781d84ef8e8ee","entry":"AllGather","repo":"TrustedLLM/AutoRegEmbed","repo_kind":"official","path":"src/modeling/modeling_autoregembed.py","file_url":"https://github.com/TrustedLLM/AutoRegEmbed/blob/HEAD/src/modeling/modeling_autoregembed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"953781d84ef8e8ee"}},{"code_sha256_prefix":"9f3a912a608f3f17","entry":"EncoderOutput","repo":"TrustedLLM/AutoRegEmbed","repo_kind":"official","path":"src/modeling/modeling_autoregembed.py","file_url":"https://github.com/TrustedLLM/AutoRegEmbed/blob/HEAD/src/modeling/modeling_autoregembed.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f3a912a608f3f17"}},{"code_sha256_prefix":"2992bbfd100d29c6","entry":"AutoRegEmbed","repo":"TrustedLLM/AutoRegEmbed","repo_kind":"official","path":"src/modeling/modeling_autoregembed.py","file_url":"https://github.com/TrustedLLM/AutoRegEmbed/blob/HEAD/src/modeling/modeling_autoregembed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2992bbfd100d29c6"}},{"code_sha256_prefix":"3c09599e24400ebe","entry":"mismatched_sizes_gather_tensor_with_grad","repo":"TrustedLLM/AutoRegEmbed","repo_kind":"official","path":"src/modeling/modeling_autoregembed.py","file_url":"https://github.com/TrustedLLM/AutoRegEmbed/blob/HEAD/src/modeling/modeling_autoregembed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c09599e24400ebe"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}