Papers › LUKE-Graph: A Transformer-based Approach with Gated Relational Graph Attention for...

LUKE-Graph: A Transformer-based Approach with Gated Relational Graph Attention for Cloze-style Reading Comprehension

12 Mar 2023arXiv:2303.06675archive 2025-07-28

Shima Foolad, Kourosh Kiani

Incorporating prior knowledge can improve existing pre-training models in cloze-style machine reading and has become a new trend in recent studies. Notably, most of the existing models have integrated external knowledge graphs (KG) and transformer-based models, such as BERT into a unified data structure. However, selecting the most relevant ambiguous entities in KG and extracting the best subgraph remains a challenge. In this paper, we propose the LUKE-Graph, a model that builds a heterogeneous graph based on the intuitive relationships between entities in a document without using any external KG. We then use a Relational Graph Attention (RGAT) network to fuse the graph's reasoning information and the contextual representation encoded by the pre-trained LUKE model. In this way, we can take advantage of LUKE, to derive an entity-aware representation; and a graph model - to exploit relation-aware representation. Moreover, we propose Gated-RGAT by augmenting RGAT with a gating mechanism that regulates the question information for the graph convolution operation. This is very similar to human reasoning processing because they always choose the best entity candidate based on the question information. Experimental results demonstrate that the LUKE-Graph achieves state-of-the-art performance on the ReCoRD dataset with commonsense reasoning.

Paper PDF

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Common Sense Reasoning Graph Attention Knowledge Graphs Machine Reading Comprehension Question Answering Reading Comprehension

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Common Sense Reasoning	ReCoRD	LUKE-Graph	EM	91.2	#8 of 45	Archive leaderboard	report
Common Sense Reasoning	ReCoRD	LUKE-Graph	F1	91.5	#8 of 45	Archive leaderboard	report
Question Answering	WikiHop	LUKE-Graph	Test	81.0	#3 of 9	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Adam Attention Attention Dropout BERT Convolution Dense Connections Dropout Layer Normalization Linear Layer Linear Warmup With Linear Decay Multi-Head Attention Residual Connection Softmax Weight Decay WordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections