</> Code with Papers by Syntology

Papers › Improving Open Information Extraction with Large Language Models: A Study on...

Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty

7 Sep 2023arXiv:2309.03433archive 2025-07-28

Chen Ling, Xujiang Zhao, Xuchao Zhang, Yanchi Liu, Wei Cheng, Haoyu Wang, Zhengzhang Chen, Takao Osaki, Katsushi Matsuda, Haifeng Chen, Liang Zhao

Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text, typically in the form of (subject, relation, object) triples. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks due to two key issues. First, LLMs struggle to distinguish irrelevant context from relevant relations and generate structured output due to the restrictions on fine-tuning the model. Second, LLMs generates responses autoregressively based on probability, which makes the predicted relations lack confidence. In this paper, we assess the capabilities of LLMs in improving the OIE task. Particularly, we propose various in-context learning strategies to enhance LLM's instruction-following ability and a demonstration uncertainty quantification module to enhance the confidence of the generated relations. Our experiments on three OIE benchmark datasets show that our approach holds its own against established supervised methods, both quantitatively and qualitatively.

PaperPDF

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

In-Context LearningInstruction FollowingOpen Information ExtractionUncertainty Quantification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Open Information Extraction CaRB GPT-3.5-Turbo w/ Selected Demo & Uncertainty F1 52.1 #9 of 29 Archive leaderboard report
Open Information Extraction CaRB LLaMA-2-70B w/ Selected Demo & Uncertainty F1 51.5 #12 of 29 Archive leaderboard report
Open Information Extraction CaRB LLaMA-2-13B w/ Selected Demo & Uncertainty F1 36.2 #26 of 29 Archive leaderboard report
Open Information Extraction OIE2016 LLaMA-2-70B w/ Selected Demo & Uncertainty F1 65.8 #6 of 12 Archive leaderboard report
Open Information Extraction OIE2016 GPT-3.5-Turbo w/ Selected Demo & Uncertainty F1 65.1 #7 of 12 Archive leaderboard report
Open Information Extraction OIE2016 LLaMA-2-13B w/ Selected Demo & Uncertainty F1 36.9 #11 of 12 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections