{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/iot-instance-wise-layer-reordering-for-1","title":"IOT: Instance-wise Layer Reordering for Transformer Structures","arxiv_id":"2103.03457","date":"2021-03-05","proceeding":"ICLR 2021 1","authors":["Jinhua Zhu","Lijun Wu","Yingce Xia","Shufang Xie","Tao Qin","Wengang Zhou","Houqiang Li","Tie-Yan Liu"],"abstract":"With sequentially stacked self-attention, (optional) encoder-decoder attention, and feed-forward layers, Transformer achieves big success in natural language processing (NLP), and many variants have been proposed. Currently, almost all these models assume that the layer order is fixed and kept the same across data samples. We observe that different data samples actually favor different orders of the layers. Based on this observation, in this work, we break the assumption of the fixed layer order in the Transformer and introduce instance-wise layer reordering into the model structure. Our Instance-wise Ordered Transformer (IOT) can model variant functions by reordered layers, which enables each sample to select the better one to improve the model performance under the constraint of almost the same number of parameters. To achieve this, we introduce a light predictor with negligible parameter and inference cost to decide the most capable and favorable layer order for any input sequence. Experiments on 3 tasks (neural machine translation, abstractive summarization, and code generation) and 9 datasets demonstrate consistent improvements of our method. We further show that our method can also be applied to other architectures beyond Transformer. Our code is released at Github.","url_abs":"https://arxiv.org/abs/2103.03457v1","url_pdf":"https://arxiv.org/pdf/2103.03457v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"iot-instance-wise-layer-reordering-for-1","repo_url":"https://github.com/instance-wise-ordered-transformer/IOT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"abstractive-text-summarization","task_name":"Abstractive Text Summarization"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.03457","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.03457"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/instance-wise-ordered-transformer/IOT","reach":null}],"summary":{"ran_draft_wrong":2,"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"96cbb5e9ca5b6be0","entry":"Embedding","repo":"instance-wise-ordered-transformer/iot","repo_kind":"official","path":"fairseq/models/transformer.py","file_url":"https://github.com/instance-wise-ordered-transformer/iot/blob/HEAD/fairseq/models/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"96cbb5e9ca5b6be0"}},{"code_sha256_prefix":"e994de3ecaaa1676","entry":"LayerNorm","repo":"instance-wise-ordered-transformer/IOT","repo_kind":"official","path":"fairseq/modules/transformer_sentence_encoder_layer.py","file_url":"https://github.com/instance-wise-ordered-transformer/IOT/blob/HEAD/fairseq/modules/transformer_sentence_encoder_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e994de3ecaaa1676"}},{"code_sha256_prefix":"8cd8cb0d1e9e63e4","entry":"Linear","repo":"instance-wise-ordered-transformer/iot","repo_kind":"official","path":"fairseq/models/transformer.py","file_url":"https://github.com/instance-wise-ordered-transformer/iot/blob/HEAD/fairseq/models/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8cd8cb0d1e9e63e4"}},{"code_sha256_prefix":"b4aacbb45af4df12","entry":"MultiheadAttention","repo":"instance-wise-ordered-transformer/IOT","repo_kind":"official","path":"fairseq/modules/transformer_sentence_encoder_layer.py","file_url":"https://github.com/instance-wise-ordered-transformer/IOT/blob/HEAD/fairseq/modules/transformer_sentence_encoder_layer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b4aacbb45af4df12"}},{"code_sha256_prefix":"b628e17c1c82b491","entry":"TransformerSentenceEncoderLayer","repo":"instance-wise-ordered-transformer/IOT","repo_kind":"official","path":"fairseq/modules/transformer_sentence_encoder_layer.py","file_url":"https://github.com/instance-wise-ordered-transformer/IOT/blob/HEAD/fairseq/modules/transformer_sentence_encoder_layer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b628e17c1c82b491"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}