Papers › Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation
Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation
Xiaohang Tang, Yi Zhou, Danushka Bollegala
Dynamic contextualised word embeddings (DCWEs) represent the temporal semantic variations of words. We propose a method for learning DCWEs by time-adapting a pretrained Masked Language Model (MLM) using time-sensitive templates. Given two snapshots C₁ and C₂ of a corpus taken respectively at two distinct timestamps T₁ and T₂, we first propose an unsupervised method to select (a) \emph{pivot} terms related to both C₁ and C₂, and (b) \emph{anchor} terms that are associated with a specific pivot term in each individual snapshot. We then generate prompts by filling manually compiled templates using the extracted pivot and anchor terms. Moreover, we propose an automatic method to learn time-sensitive templates from C₁ and C₂, without requiring any human supervision. Next, we use the generated prompts to adapt a pretrained MLM to T₂ by fine-tuning using those prompts. Multiple experiments show that our proposed method reduces the perplexity of test sentences in C₂, outperforming the current state-of-the-art.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections