Papers › ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning

ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning

30 Dec 2020EMNLP 2021 11arXiv:2012.15283archive 2025-07-28

Rujun Han, Xiang Ren, Nanyun Peng

While pre-trained language models (PTLMs) have achieved noticeable success on many NLP tasks, they still struggle for tasks that require event temporal reasoning, which is essential for event-centric applications. We present a continual pre-training approach that equips PTLMs with targeted knowledge about event temporal relations. We design self-supervised learning objectives to recover masked-out event and temporal indicators and to discriminate sentences from their corrupted counterparts (where event or temporal indicators got replaced). By further pre-training a PTLM with these objectives jointly, we reinforce its attention to event and temporal information, yielding enhanced capability on event temporal reasoning. This effective continual pre-training framework for event temporal reasoning (ECONET) improves the PTLMs' fine-tuning performances across five relation extraction and question answering tasks and achieves new or on-par state-of-the-art performances in most of our downstream tasks.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

pluslabnlp/econet officialmentioned in paperpytorch report

ZHEvent/ZHEvent.github.io mentioned on GitHubMIT report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Continual Pretraining Language Modelling Machine Reading Comprehension Question Answering Reading Comprehension Relation Extraction Self-Supervised Learning

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Question Answering	Torque	ECONET	C	37.0	#1 of 2	Archive leaderboard	report
Question Answering	Torque	ECONET	EM	52.0	#1 of 2	Archive leaderboard	report
Question Answering	Torque	ECONET	F1	76.3	#1 of 2	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Adam Attention Attention Dropout BERT Dense Connections Dropout ELECTRA Layer Normalization Linear Layer Linear Warmup With Linear Decay Multi-Head Attention Residual Connection RoBERTa Softmax Weight Decay WordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections