{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/will-llms-replace-the-encoder-only-models-in","title":"Will LLMs Replace the Encoder-Only Models in Temporal Relation Classification?","arxiv_id":"2410.10476","date":"2024-10-14","proceeding":null,"authors":["Gabriel Roccabruna","Massimo Rizzoli","Giuseppe Riccardi"],"abstract":"The automatic detection of temporal relations among events has been mainly investigated with encoder-only models such as RoBERTa. Large Language Models (LLM) have recently shown promising performance in temporal reasoning tasks such as temporal question answering. Nevertheless, recent studies have tested the LLMs' performance in detecting temporal relations of closed-source models only, limiting the interpretability of those results. In this work, we investigate LLMs' performance and decision process in the Temporal Relation Classification task. First, we assess the performance of seven open and closed-sourced LLMs experimenting with in-context learning and lightweight fine-tuning approaches. Results show that LLMs with in-context learning significantly underperform smaller encoder-only models based on RoBERTa. Then, we delve into the possible reasons for this gap by applying explainable methods. The outcome suggests a limitation of LLMs in this task due to their autoregressive nature, which causes them to focus only on the last part of the sequence. Additionally, we evaluate the word embeddings of these two models to better understand their pre-training differences. The code and the fine-tuned models can be found respectively on GitHub.","url_abs":"https://arxiv.org/abs/2410.10476v2","url_pdf":"https://arxiv.org/pdf/2410.10476v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"will-llms-replace-the-encoder-only-models-in","repo_url":"https://github.com/brownfortress/llms-trc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"relation-classification","task_name":"Relation Classification"},{"task_slug":"temporal-relation-classification","task_name":"Temporal Relation Classification"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"roberta","method_name":"RoBERTa"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.10476","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.10476"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/BrownFortress/LLMs-TRC","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4e80b769da0a37b8","entry":"aggregate","repo":"BrownFortress/LLMs-TRC","repo_kind":"official","path":"XAI_analysis/utils/attribution_scores.py","file_url":"https://github.com/BrownFortress/LLMs-TRC/blob/HEAD/XAI_analysis/utils/attribution_scores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e80b769da0a37b8"}},{"code_sha256_prefix":"649a696f78a75f71","entry":"construct_references","repo":"BrownFortress/LLMs-TRC","repo_kind":"official","path":"XAI_analysis/utils/attribution_scores.py","file_url":"https://github.com/BrownFortress/LLMs-TRC/blob/HEAD/XAI_analysis/utils/attribution_scores.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"649a696f78a75f71"}},{"code_sha256_prefix":"b2c9d8d4ae54a54b","entry":"get_result_by_id","repo":"BrownFortress/LLMs-TRC","repo_kind":"official","path":"ICL_and_FT/stat_reader.py","file_url":"https://github.com/BrownFortress/LLMs-TRC/blob/HEAD/ICL_and_FT/stat_reader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2c9d8d4ae54a54b"}},{"code_sha256_prefix":"166eaed4a0806704","entry":"has_numbers","repo":"BrownFortress/LLMs-TRC","repo_kind":"official","path":"ICL_and_FT/stat_reader.py","file_url":"https://github.com/BrownFortress/LLMs-TRC/blob/HEAD/ICL_and_FT/stat_reader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"166eaed4a0806704"}},{"code_sha256_prefix":"0f40997a01ccd3cc","entry":"is_there_timex","repo":"BrownFortress/LLMs-TRC","repo_kind":"official","path":"ICL_and_FT/stat_reader.py","file_url":"https://github.com/BrownFortress/LLMs-TRC/blob/HEAD/ICL_and_FT/stat_reader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f40997a01ccd3cc"}},{"code_sha256_prefix":"ded0f385824cb4b2","entry":"aggregateLLAMA","repo":"BrownFortress/LLMs-TRC","repo_kind":"official","path":"XAI_analysis/utils/attribution_scores.py","file_url":"https://github.com/BrownFortress/LLMs-TRC/blob/HEAD/XAI_analysis/utils/attribution_scores.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ded0f385824cb4b2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}