{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sequencer-deep-lstm-for-image-classification","title":"Sequencer: Deep LSTM for Image Classification","arxiv_id":"2205.01972","date":"2022-05-04","proceeding":null,"authors":["Yuki Tatsunami","Masato Taki"],"abstract":"In recent computer vision research, the advent of the Vision Transformer (ViT) has rapidly revolutionized various architectural design efforts: ViT achieved state-of-the-art image classification performance using self-attention found in natural language processing, and MLP-Mixer achieved competitive performance using simple multi-layer perceptrons. In contrast, several studies have also suggested that carefully redesigned convolutional neural networks (CNNs) can achieve advanced performance comparable to ViT without resorting to these new ideas. Against this background, there is growing interest in what inductive bias is suitable for computer vision. Here we propose Sequencer, a novel and competitive architecture alternative to ViT that provides a new perspective on these issues. Unlike ViTs, Sequencer models long-range dependencies using LSTMs rather than self-attention layers. We also propose a two-dimensional version of Sequencer module, where an LSTM is decomposed into vertical and horizontal LSTMs to enhance performance. Despite its simplicity, several experiments demonstrate that Sequencer performs impressively well: Sequencer2D-L, with 54M parameters, realizes 84.6% top-1 accuracy on only ImageNet-1K. Not only that, we show that it has good transferability and the robust resolution adaptability on double resolution-band.","url_abs":"https://arxiv.org/abs/2205.01972v4","url_pdf":"https://arxiv.org/pdf/2205.01972v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sequencer-deep-lstm-for-image-classification","repo_url":"https://github.com/okojoalg/sequencer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"sequencer-deep-lstm-for-image-classification","repo_url":"https://github.com/rwightman/pytorch-image-models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"sequencer-deep-lstm-for-image-classification","repo_url":"https://github.com/liuruiyang98/Jittor-MLP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"sequencer-deep-lstm-for-image-classification","repo_url":"https://github.com/timeseriesAI/tsai","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"sequencer-deep-lstm-for-image-classification","repo_url":"https://gitlab.com/birder/birder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"adamw","method_name":"AdamW"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"bilstm","method_name":"BiLSTM"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"mlp-mixer","method_name":"MLP-Mixer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/domain-generalization-on-imagenet-a","task":"Domain Generalization","dataset":"ImageNet-A","model":"Sequencer2D-L","rank_in_archive_order":25,"of":39,"metrics":{"Top-1 accuracy %":"35.5"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-imagenet-c","task":"Domain Generalization","dataset":"ImageNet-C","model":"Sequencer2D-L","rank_in_archive_order":25,"of":47,"metrics":{"mean Corruption Error (mCE)":"48.9"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-imagenet-r","task":"Domain Generalization","dataset":"ImageNet-R","model":"Sequencer2D-L","rank_in_archive_order":27,"of":39,"metrics":{"Top-1 Error Rate":"51.9"},"uses_additional_data":false},{"leaderboard":"/sota/domain-generalization-on-imagenet-sketch","task":"Domain Generalization","dataset":"ImageNet-Sketch","model":"Sequencer2D-L","rank_in_archive_order":20,"of":20,"metrics":{"Top-1 accuracy":"35.8"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"Sequencer2D-L↑392","rank_in_archive_order":303,"of":1060,"metrics":{"GFLOPs":"50.7","Number of params":"54M","Top 1 Accuracy":"84.6%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"Sequencer2D-L","rank_in_archive_order":432,"of":1060,"metrics":{"GFLOPs":"16.6","Number of params":"54M","Top 1 Accuracy":"83.4%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"Sequencer2D-M","rank_in_archive_order":500,"of":1060,"metrics":{"GFLOPs":"11.1","Number of params":"38M","Top 1 Accuracy":"82.8%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"Sequencer2D-S","rank_in_archive_order":552,"of":1060,"metrics":{"GFLOPs":"8.4","Number of params":"28M","Top 1 Accuracy":"82.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet-real","task":"Image Classification","dataset":"ImageNet ReaL","model":"Sequencer2D-L","rank_in_archive_order":29,"of":57,"metrics":{"Accuracy":"87.9"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet-v2","task":"Image Classification","dataset":"ImageNet V2","model":"Sequencer2D-L","rank_in_archive_order":21,"of":33,"metrics":{"Top 1 Accuracy":"73.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.01972","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.01972"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rwightman/pytorch-image-models","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/okojoalg/sequencer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liuruiyang98/Jittor-MLP","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/timeseriesAI/tsai","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://gitlab.com/birder/birder","reach":null}],"summary":{"ran":4,"unverified":5},"by_repo_kind":{"official":{"samples":9,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"71520c22fcd16235","entry":"Downsample2D","repo":"okojoalg/sequencer","repo_kind":"official","path":"models/two_dim_sequencer.py","file_url":"https://github.com/okojoalg/sequencer/blob/HEAD/models/two_dim_sequencer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"71520c22fcd16235"}},{"code_sha256_prefix":"d21dc0c3ff2326b2","entry":"PatchEmbed","repo":"okojoalg/sequencer","repo_kind":"official","path":"models/two_dim_sequencer.py","file_url":"https://github.com/okojoalg/sequencer/blob/HEAD/models/two_dim_sequencer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d21dc0c3ff2326b2"}},{"code_sha256_prefix":"b68d5c2770b76e3e","entry":"RNN2DBase","repo":"okojoalg/sequencer","repo_kind":"official","path":"models/two_dim_sequencer.py","file_url":"https://github.com/okojoalg/sequencer/blob/HEAD/models/two_dim_sequencer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b68d5c2770b76e3e"}},{"code_sha256_prefix":"cee0eb3911fd1719","entry":"RNNIdentity","repo":"okojoalg/sequencer","repo_kind":"official","path":"models/two_dim_sequencer.py","file_url":"https://github.com/okojoalg/sequencer/blob/HEAD/models/two_dim_sequencer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cee0eb3911fd1719"}},{"code_sha256_prefix":"e5ad67deb204c7fb","entry":"LSTM2D","repo":"okojoalg/sequencer","repo_kind":"official","path":"models/two_dim_sequencer.py","file_url":"https://github.com/okojoalg/sequencer/blob/HEAD/models/two_dim_sequencer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5ad67deb204c7fb"}},{"code_sha256_prefix":"07c6c53559ef34c9","entry":"Sequencer2D","repo":"okojoalg/sequencer","repo_kind":"official","path":"models/two_dim_sequencer.py","file_url":"https://github.com/okojoalg/sequencer/blob/HEAD/models/two_dim_sequencer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"07c6c53559ef34c9"}},{"code_sha256_prefix":"21b70a992bd5f441","entry":"Sequencer2DBlock","repo":"okojoalg/sequencer","repo_kind":"official","path":"models/two_dim_sequencer.py","file_url":"https://github.com/okojoalg/sequencer/blob/HEAD/models/two_dim_sequencer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"21b70a992bd5f441"}},{"code_sha256_prefix":"58c0c12066377b33","entry":"_init_weights","repo":"okojoalg/sequencer","repo_kind":"official","path":"models/two_dim_sequencer.py","file_url":"https://github.com/okojoalg/sequencer/blob/HEAD/models/two_dim_sequencer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"58c0c12066377b33"}},{"code_sha256_prefix":"caa022d5744fcda0","entry":"get_stage","repo":"okojoalg/sequencer","repo_kind":"official","path":"models/two_dim_sequencer.py","file_url":"https://github.com/okojoalg/sequencer/blob/HEAD/models/two_dim_sequencer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"caa022d5744fcda0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}