{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/time-aware-large-kernel-convolutions","title":"Time-aware Large Kernel Convolutions","arxiv_id":"2002.03184","date":"2020-02-08","proceeding":"ICML 2020 1","authors":["Vasileios Lioutas","Yuhong Guo"],"abstract":"To date, most state-of-the-art sequence modeling architectures use attention to build generative models for language based tasks. Some of these models use all the available sequence tokens to generate an attention distribution which results in time complexity of $O(n^2)$. Alternatively, they utilize depthwise convolutions with softmax normalized kernels of size $k$ acting as a limited-window self-attention, resulting in time complexity of $O(k{\\cdot}n)$. In this paper, we introduce Time-aware Large Kernel (TaLK) Convolutions, a novel adaptive convolution operation that learns to predict the size of a summation kernel instead of using a fixed-sized kernel matrix. This method yields a time complexity of $O(n)$, effectively making the sequence encoding process linear to the number of tokens. We evaluate the proposed method on large-scale standard machine translation, abstractive summarization and language modeling datasets and show that TaLK Convolutions constitute an efficient improvement over other attention/convolution based approaches.","url_abs":"https://arxiv.org/abs/2002.03184v2","url_pdf":"https://arxiv.org/pdf/2002.03184v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"time-aware-large-kernel-convolutions","repo_url":"https://github.com/lioutasb/TaLKConvolutions","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"document-summarization","task_name":"Document Summarization"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"talk-convolutions","method_name":"TaLK Convolution"}],"datasets_introduced":[],"methods_introduced":[{"slug":"talk-convolutions","name":"TaLK Convolution","full_name":"Time-aware Large Kernel Convolution"}],"results":[{"leaderboard":"/sota/document-summarization-on-cnn-daily-mail","task":"Document Summarization","dataset":"CNN / Daily Mail","model":"TaLK Convolutions (Deep)","rank_in_archive_order":17,"of":26,"metrics":{"ROUGE-1":"40.59","ROUGE-2":"18.97","ROUGE-L":"36.81"},"uses_additional_data":false},{"leaderboard":"/sota/document-summarization-on-cnn-daily-mail","task":"Document Summarization","dataset":"CNN / Daily Mail","model":"TaLK Convolutions (Standard)","rank_in_archive_order":19,"of":26,"metrics":{"ROUGE-1":"40.03","ROUGE-2":"18.45","ROUGE-L":"36.13"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"TaLK Convolutions","rank_in_archive_order":51,"of":89,"metrics":{"Number of params":"240M","Test perplexity":"23.3"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-iwslt2014-german","task":"Machine Translation","dataset":"IWSLT2014 German-English","model":"TaLK Convolutions","rank_in_archive_order":21,"of":34,"metrics":{"BLEU score":"35.5"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-french","task":"Machine Translation","dataset":"WMT2014 English-French","model":"TaLK Convolutions","rank_in_archive_order":14,"of":57,"metrics":{"BLEU score":"43.2"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"TaLK Convolutions","rank_in_archive_order":20,"of":91,"metrics":{"BLEU score":"29.6","Number of Params":"209M"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2002.03184","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2002.03184"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/lioutasb/TaLKConvolutions","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":4,"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"96cbb5e9ca5b6be0","entry":"Embedding","repo":"lioutasb/TaLKConvolutions","repo_kind":"official","path":"talkconv/talkconv_fairseq/talkconv.py","file_url":"https://github.com/lioutasb/TaLKConvolutions/blob/HEAD/talkconv/talkconv_fairseq/talkconv.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96cbb5e9ca5b6be0"}},{"code_sha256_prefix":"8cd8cb0d1e9e63e4","entry":"Linear","repo":"lioutasb/TaLKConvolutions","repo_kind":"official","path":"talkconv/talkconv_fairseq/talkconv.py","file_url":"https://github.com/lioutasb/TaLKConvolutions/blob/HEAD/talkconv/talkconv_fairseq/talkconv.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cd8cb0d1e9e63e4"}},{"code_sha256_prefix":"93ba0aaaba0bfada","entry":"average_checkpoints","repo":"lioutasb/talkconvolutions","repo_kind":"official","path":"utils/average_checkpoints.py","file_url":"https://github.com/lioutasb/talkconvolutions/blob/HEAD/utils/average_checkpoints.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93ba0aaaba0bfada"}},{"code_sha256_prefix":"8dd1def34e4485c2","entry":"last_n_checkpoints","repo":"lioutasb/talkconvolutions","repo_kind":"official","path":"utils/average_checkpoints.py","file_url":"https://github.com/lioutasb/talkconvolutions/blob/HEAD/utils/average_checkpoints.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dd1def34e4485c2"}},{"code_sha256_prefix":"99de76104bad1fc9","entry":"last_n_checkpoints","repo":"lioutasb/TaLKConvolutions","repo_kind":"official","path":"utils/average_checkpoints.py","file_url":"https://github.com/lioutasb/TaLKConvolutions/blob/HEAD/utils/average_checkpoints.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99de76104bad1fc9"}},{"code_sha256_prefix":"7cc2e301e30389d1","entry":"average_checkpoints","repo":"lioutasb/TaLKConvolutions","repo_kind":"official","path":"utils/average_checkpoints.py","file_url":"https://github.com/lioutasb/TaLKConvolutions/blob/HEAD/utils/average_checkpoints.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cc2e301e30389d1"}},{"code_sha256_prefix":"bddb1c75a5038e42","entry":"gelu","repo":"lioutasb/TaLKConvolutions","repo_kind":"official","path":"talkconv/talkconv_fairseq/talkconv.py","file_url":"https://github.com/lioutasb/TaLKConvolutions/blob/HEAD/talkconv/talkconv_fairseq/talkconv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bddb1c75a5038e42"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}