{"url":"/method/etc","slug":"etc","name":"ETC","full_name":"Extended Transformer Construction","full_name_withheld":false,"description_markdown":"**Extended Transformer Construction**, or **ETC**, is an extension of the [Transformer](https://paperswithcode.com/method/transformer) architecture with a new attention mechanism that extends the original in two main ways: (1) it allows scaling up the input length from 512 to several thousands; and (2) it can ingesting structured inputs instead of just linear sequences. The key ideas that enable ETC to achieve these are a new [global-local attention mechanism](https://paperswithcode.com/method/global-local-attention), coupled with [relative position encodings](https://paperswithcode.com/method/relative-position-encodings). ETC also allows lifting weights from existing [BERT](https://paperswithcode.com/method/bert) models, saving computational resources while training.","description_state":"present","introduced_year":null,"introduced_by":{"title":"ETC: Encoding Long and Structured Inputs in Transformers","paper":"/paper/etc-encoding-long-and-structured-data-in","first_author":"Joshua Ainslie","n_authors":10,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/etc-encoding-long-and-structured-data-in"},"source":{"url":"https://arxiv.org/abs/2004.08483v5","title":"ETC: Encoding Long and Structured Inputs in Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]}],"n_papers_tagged":36,"archive_num_papers":36,"papers_newest_first":[{"paper":null,"title":"Two-Player Zero-Sum Games with Bandit Feedback","date":"2025-06-17","arxiv_id":"2506.14518","n_code_links":0,"syntology":null},{"paper":null,"title":"Koopman-Based Event-Triggered Control from Data","date":"2025-04-19","arxiv_id":"2504.14334","n_code_links":0,"syntology":null},{"paper":null,"title":"Performance-Barrier Event-Triggered PDE Control of Traffic Flow","date":"2025-01-01","arxiv_id":"2501.00722","n_code_links":0,"syntology":null},{"paper":null,"title":"Automated Toll Management System Using RFID and Image Processing","date":"2024-12-02","arxiv_id":"2412.01728","n_code_links":0,"syntology":null},{"paper":null,"title":"Hierarchical Event-Triggered Systems: Safe Learning of Quasi-Optimal Deadline Policies","date":"2024-09-15","arxiv_id":"2409.09812","n_code_links":0,"syntology":null},{"paper":null,"title":"Performance-Barrier Event-Triggered Control of a Class of Reaction-Diffusion PDEs","date":"2024-07-11","arxiv_id":"2407.08178","n_code_links":0,"syntology":null},{"paper":null,"title":"Contextual Dynamic Pricing: Algorithms, Optimality, and Local Differential Privacy Constraints","date":"2024-06-04","arxiv_id":"2406.02424","n_code_links":0,"syntology":null},{"paper":null,"title":"Event-Triggered Robust Cooperative Output Regulation for a Class of Linear Multi-Agent Systems with an Unknown Exosystem","date":"2024-03-01","arxiv_id":"2403.00645","n_code_links":0,"syntology":null},{"paper":null,"title":"Replication-proof Bandit Mechanism Design with Bayesian Agents","date":"2023-12-28","arxiv_id":"2312.16896","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning-based Scheduling for Information Accuracy and Freshness in Wireless Networks","date":"2023-10-24","arxiv_id":"2310.15705","n_code_links":0,"syntology":null},{"paper":null,"title":"Listen to Minority: Encrypted Traffic Classification for Class Imbalance with Contrastive Pre-Training","date":"2023-08-31","arxiv_id":"2308.16453","n_code_links":0,"syntology":null},{"paper":null,"title":"High-dimensional Contextual Bandit Problem without Sparsity","date":"2023-06-19","arxiv_id":"2306.11017","n_code_links":0,"syntology":null},{"paper":null,"title":"Permutation Decision Trees","date":"2023-06-05","arxiv_id":"2306.02617","n_code_links":0,"syntology":null},{"paper":null,"title":"An Improved Heart Disease Prediction Using Stacked Ensemble Method","date":"2023-04-12","arxiv_id":"2304.06015","n_code_links":0,"syntology":null},{"paper":null,"title":"Asynchronous Event-Triggered Control for Non-Linear Systems","date":"2022-11-25","arxiv_id":"2211.13846","n_code_links":0,"syntology":null},{"paper":null,"title":"LittleBird: Efficient Faster & Longer Transformer for Question Answering","date":"2022-10-21","arxiv_id":"2210.11870","n_code_links":0,"syntology":null},{"paper":null,"title":"Short Text Pre-training with Extended Token Classification for E-commerce Query Understanding","date":"2022-10-08","arxiv_id":"2210.03915","n_code_links":0,"syntology":null},{"paper":null,"title":"Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiency","date":"2022-05-26","arxiv_id":"2205.13476","n_code_links":0,"syntology":null},{"paper":"/paper/thompson-sampling-for-bandit-learning-in","title":"Thompson Sampling for Bandit Learning in Matching Markets","date":"2022-04-26","arxiv_id":"2204.12048","n_code_links":1,"syntology":null},{"paper":null,"title":"ETCetera: beyond Event-Triggered Control","date":"2022-03-03","arxiv_id":"2203.01623","n_code_links":0,"syntology":null},{"paper":null,"title":"Formal Analysis of the Sampling Behaviour of Stochastic Event-Triggered Control","date":"2022-02-21","arxiv_id":"2202.10178","n_code_links":0,"syntology":null},{"paper":null,"title":"Chaos and order in event-triggered control","date":"2022-01-12","arxiv_id":"2201.04462","n_code_links":0,"syntology":null},{"paper":null,"title":"Detecting Extratropical Cyclones of the Northern Hemisphere with Single Shot Detector","date":"2021-12-01","arxiv_id":"2112.01283","n_code_links":0,"syntology":null},{"paper":null,"title":"Bandits with Dynamic Arm-acquisition Costs","date":"2021-10-23","arxiv_id":"2110.12118","n_code_links":0,"syntology":null},{"paper":null,"title":"Computing the average inter-sample time of event-triggered control using quantitative automata","date":"2021-09-29","arxiv_id":"2109.14391","n_code_links":0,"syntology":null},{"paper":null,"title":"Uncertainties and output feedback in rollout event-triggered control","date":"2021-08-20","arxiv_id":"2108.09125","n_code_links":0,"syntology":null},{"paper":null,"title":"Rollout event-triggered control: reconciling event- and time-triggered control","date":"2021-08-06","arxiv_id":"2108.02994","n_code_links":0,"syntology":null},{"paper":null,"title":"Debiasing Samples from Online Learning Using Bootstrap","date":"2021-07-31","arxiv_id":"2108.00236","n_code_links":0,"syntology":null},{"paper":null,"title":"Model Selection for Generic Contextual Bandits","date":"2021-07-07","arxiv_id":"2107.03455","n_code_links":0,"syntology":null},{"paper":null,"title":"Abstracting the Sampling Behaviour of Stochastic Linear Periodic Event-Triggered Control Systems","date":"2021-03-25","arxiv_id":"2103.13839","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/scheduling","name":"Scheduling","papers":3},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":"/task/multi-armed-bandits","name":"Multi-Armed Bandits","papers":2},{"task":"/task/question-answering","name":"Question Answering","papers":2},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":2},{"task":"/task/thompson-sampling","name":"Thompson Sampling","papers":2},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":2},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":1},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":1},{"task":"/task/diagnostic","name":"Diagnostic","papers":1},{"task":"/task/disease-prediction","name":"Disease Prediction","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/linguistic-acceptability","name":"Linguistic Acceptability","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/management","name":"Management","papers":1},{"task":"/task/masked-language-modeling","name":"Masked Language Modeling","papers":1},{"task":"/task/model-predictive-control","name":"Model Predictive Control","papers":1},{"task":"/task/model-selection","name":"Model Selection","papers":1},{"task":"/task/natural-language-inference","name":"Natural Language Inference","papers":1},{"task":"/task/natural-questions","name":"Natural Questions","papers":1}],"tasks_shown":20,"n_tasks":38,"usage_by_year":[{"year":"2020","papers":5},{"year":"2021","papers":9},{"year":"2022","papers":8},{"year":"2023","papers":6},{"year":"2024","papers":5},{"year":"2025","papers":3}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/etc"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}