{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/seqpe-transformer-with-sequential-position","title":"SeqPE: Transformer with Sequential Position Encoding","arxiv_id":"2506.13277","date":"2025-06-16","proceeding":null,"authors":["Huayang Li","Yahui Liu","Hongyu Sun","Deng Cai","Leyang Cui","Wei Bi","Peilin Zhao","Taro Watanabe"],"abstract":"Since self-attention layers in Transformers are permutation invariant by design, positional encodings must be explicitly incorporated to enable spatial understanding. However, fixed-size lookup tables used in traditional learnable position embeddings (PEs) limit extrapolation capabilities beyond pre-trained sequence lengths. Expert-designed methods such as ALiBi and RoPE, mitigate this limitation but demand extensive modifications for adapting to new modalities, underscoring fundamental challenges in adaptability and scalability. In this work, we present SeqPE, a unified and fully learnable position encoding framework that represents each $n$-dimensional position index as a symbolic sequence and employs a lightweight sequential position encoder to learn their embeddings in an end-to-end manner. To regularize SeqPE's embedding space, we introduce two complementary objectives: a contrastive objective that aligns embedding distances with a predefined position-distance function, and a knowledge distillation loss that anchors out-of-distribution position embeddings to in-distribution teacher representations, further enhancing extrapolation performance. Experiments across language modeling, long-context question answering, and 2D image classification demonstrate that SeqPE not only surpasses strong baselines in perplexity, exact match (EM), and accuracy--particularly under context length extrapolation--but also enables seamless generalization to multi-dimensional inputs without requiring manual architectural redesign. We release our code, data, and checkpoints at https://github.com/ghrua/seqpe.","url_abs":"https://arxiv.org/abs/2506.13277v2","url_pdf":"https://arxiv.org/pdf/2506.13277v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"seqpe-transformer-with-sequential-position","repo_url":"https://github.com/ghrua/seqpe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":null,"task_name":"Position"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"alibi","method_name":"ALiBi"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.13277","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.13277"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ghrua/seqpe","reach":{"status":"ok"}}],"summary":{"ran_violates":2,"ran_draft_wrong":2,"unverified":8},"by_repo_kind":{"official":{"samples":12,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"89c2cf3b97685684","entry":"is_image_file","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/eval/prepare_imagenet_c.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/eval/prepare_imagenet_c.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89c2cf3b97685684"}},{"code_sha256_prefix":"7a93b4bf9b9553b9","entry":"auc","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/eval/cal_imagenet_c.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/eval/cal_imagenet_c.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a93b4bf9b9553b9"}},{"code_sha256_prefix":"9347fb42c15d6687","entry":"find_classes","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/eval/prepare_imagenet_c.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/eval/prepare_imagenet_c.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9347fb42c15d6687"}},{"code_sha256_prefix":"87889f77c597efa2","entry":"make_dataset","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/eval/prepare_imagenet_c.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/eval/prepare_imagenet_c.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87889f77c597efa2"}},{"code_sha256_prefix":"165c5336f89f847c","entry":"agreement","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/consistency.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/consistency.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"165c5336f89f847c"}},{"code_sha256_prefix":"c02cc4e0b5720211","entry":"build_scheduler","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/lr_scheduler.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/lr_scheduler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c02cc4e0b5720211"}},{"code_sha256_prefix":"c62cad01937c7c94","entry":"conver_to_numpy","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/models/analysis_utils.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/models/analysis_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c62cad01937c7c94"}},{"code_sha256_prefix":"80450600f7f09b0a","entry":"create_logger","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/logger.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80450600f7f09b0a"}},{"code_sha256_prefix":"a4c21041d0740a79","entry":"get_config","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/config.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4c21041d0740a79"}},{"code_sha256_prefix":"e2db69c3a1bd384a","entry":"grad_loss_breaching","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/gradvit/gradvit_losses.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/gradvit/gradvit_losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2db69c3a1bd384a"}},{"code_sha256_prefix":"04948c144e3d948f","entry":"image_loss","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/gradvit/gradvit_losses.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/gradvit/gradvit_losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04948c144e3d948f"}},{"code_sha256_prefix":"4cd3558621f0286a","entry":"l2_norm_anl","repo":"ghrua/seqpe","repo_kind":"official","path":"image_seqpe/models/analysis_utils.py","file_url":"https://github.com/ghrua/seqpe/blob/HEAD/image_seqpe/models/analysis_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4cd3558621f0286a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}