{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/augmenting-transformers-with-recursively","title":"Augmenting Transformers with Recursively Composed Multi-grained Representations","arxiv_id":"2309.16319","date":"2023-09-28","proceeding":null,"authors":["Xiang Hu","Qingyang Zhu","Kewei Tu","Wei Wu"],"abstract":"We present ReCAT, a recursive composition augmented Transformer that is able to explicitly model hierarchical syntactic structures of raw texts without relying on gold trees during both learning and inference. Existing research along this line restricts data to follow a hierarchical tree structure and thus lacks inter-span communications. To overcome the problem, we propose a novel contextual inside-outside (CIO) layer that learns contextualized representations of spans through bottom-up and top-down passes, where a bottom-up pass forms representations of high-level spans by composing low-level spans, while a top-down pass combines information inside and outside a span. By stacking several CIO layers between the embedding layer and the attention layers in Transformer, the ReCAT model can perform both deep intra-span and deep inter-span interactions, and thus generate multi-grained representations fully contextualized with other spans. Moreover, the CIO layers can be jointly pre-trained with Transformers, making ReCAT enjoy scaling ability, strong performance, and interpretability at the same time. We conduct experiments on various sentence-level and span-level tasks. Evaluation results indicate that ReCAT can significantly outperform vanilla Transformer models on all span-level tasks and baselines that combine recursive networks with Transformers on natural language inference tasks. More interestingly, the hierarchical structures induced by ReCAT exhibit strong consistency with human-annotated syntactic trees, indicating good interpretability brought by the CIO layers.","url_abs":"https://arxiv.org/abs/2309.16319v2","url_pdf":"https://arxiv.org/pdf/2309.16319v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"augmenting-transformers-with-recursively","repo_url":"https://github.com/ant-research/structuredlm_rtdt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"constituency-grammar-induction","task_name":"Constituency Grammar Induction"},{"task_slug":"natural-language-inference","task_name":"Natural Language Inference"},{"task_slug":"semantic-role-labeling","task_name":"Semantic Role Labeling"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/constituency-grammar-induction-on-ptb","task":"Constituency Grammar Induction","dataset":"PTB Diagnostic ECG Database","model":"ReCAT","rank_in_archive_order":6,"of":24,"metrics":{"Mean F1 (WSJ)":"65.0"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-role-labeling-on-ontonotes","task":"Semantic Role Labeling","dataset":"OntoNotes","model":"ReCAT(pretrained on wikitext103)","rank_in_archive_order":4,"of":17,"metrics":{"F1":"88.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.16319","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.16319"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ant-research/structuredlm_rtdt","reach":null}],"summary":{"ran":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5d098ec6c3be99a0","entry":"InsideEncoder","repo":"ant-research/structuredlm_rtdt","repo_kind":"official","path":"model/tree_encoder.py","file_url":"https://github.com/ant-research/structuredlm_rtdt/blob/HEAD/model/tree_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d098ec6c3be99a0"}},{"code_sha256_prefix":"ef2b7ab2ebb4fc06","entry":"TreeEncoderLayer","repo":"ant-research/structuredlm_rtdt","repo_kind":"official","path":"model/tree_encoder.py","file_url":"https://github.com/ant-research/structuredlm_rtdt/blob/HEAD/model/tree_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef2b7ab2ebb4fc06"}},{"code_sha256_prefix":"a9d317f927330280","entry":"_get_activation_fn","repo":"ant-research/structuredlm_rtdt","repo_kind":"official","path":"model/tree_encoder.py","file_url":"https://github.com/ant-research/structuredlm_rtdt/blob/HEAD/model/tree_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a9d317f927330280"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}