{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pixel-sentence-representation-learning","title":"Pixel Sentence Representation Learning","arxiv_id":"2402.08183","date":"2024-02-13","proceeding":null,"authors":["Chenghao Xiao","Zhuoxu Huang","Danlu Chen","G Thomas Hudson","Yizhi Li","Haoran Duan","Chenghua Lin","Jie Fu","Jungong Han","Noura Al Moubayed"],"abstract":"Pretrained language models are long known to be subpar in capturing sentence and document-level semantics. Though heavily investigated, transferring perturbation-based methods from unsupervised visual representation learning to NLP remains an unsolved problem. This is largely due to the discreteness of subword units brought by tokenization of language models, limiting small perturbations of inputs to form semantics-preserved positive pairs. In this work, we conceptualize the learning of sentence-level textual semantics as a visual representation learning process. Drawing from cognitive and linguistic sciences, we introduce an unsupervised visual sentence representation learning framework, employing visually-grounded text perturbation methods like typos and word order shuffling, resonating with human cognitive patterns, and enabling perturbation to texts to be perceived as continuous. Our approach is further bolstered by large-scale unsupervised topical alignment training and natural language inference supervision, achieving comparable performance in semantic textual similarity (STS) to existing state-of-the-art NLP methods. Additionally, we unveil our method's inherent zero-shot cross-lingual transferability and a unique leapfrogging pattern across languages during iterative training. To our knowledge, this is the first representation learning method devoid of traditional language models for understanding sentence and document semantics, marking a stride closer to human-like textual comprehension. Our code is available at https://github.com/gowitheflow-1998/Pixel-Linguist","url_abs":"https://arxiv.org/abs/2402.08183v1","url_pdf":"https://arxiv.org/pdf/2402.08183v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pixel-sentence-representation-learning","repo_url":"https://github.com/gowitheflow-1998/pixel-linguist","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"pixel-sentence-representation-learning","repo_url":"https://github.com/zhh6425/MotionPointNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"sts","task_name":"STS"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.08183","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.08183"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gowitheflow-1998/pixel-linguist","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhh6425/MotionPointNet","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"b14b24b4d703c09c","entry":"apply_self_attention_rules","repo":"gowitheflow-1998/pixel-linguist","repo_kind":"official","path":"src/pixel/utils/interpretability.py","file_url":"https://github.com/gowitheflow-1998/pixel-linguist/blob/HEAD/src/pixel/utils/interpretability.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b14b24b4d703c09c"}},{"code_sha256_prefix":"57427ec7f97422a9","entry":"avg_heads","repo":"gowitheflow-1998/pixel-linguist","repo_kind":"official","path":"src/pixel/utils/interpretability.py","file_url":"https://github.com/gowitheflow-1998/pixel-linguist/blob/HEAD/src/pixel/utils/interpretability.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57427ec7f97422a9"}},{"code_sha256_prefix":"21b2990d1b908a39","entry":"glue_strip_spaces","repo":"gowitheflow-1998/pixel-linguist","repo_kind":"official","path":"src/pixel/utils/misc.py","file_url":"https://github.com/gowitheflow-1998/pixel-linguist/blob/HEAD/src/pixel/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21b2990d1b908a39"}},{"code_sha256_prefix":"62adb307e70b5e19","entry":"generate_relevance","repo":"gowitheflow-1998/pixel-linguist","repo_kind":"official","path":"src/pixel/utils/interpretability.py","file_url":"https://github.com/gowitheflow-1998/pixel-linguist/blob/HEAD/src/pixel/utils/interpretability.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62adb307e70b5e19"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}