Papers › Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

4 Sep 2018EMNLP 2018 10arXiv:1809.00786archive 2025-07-28

Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, Yoav Artzi

We propose to decompose instruction execution to goal prediction and action generation. We design a model that maps raw visual observations to goals using LINGUNET, a language-conditioned image generation network, and then generates the actions required to complete them. Our model is trained from demonstration only without external resources. To evaluate our approach, we introduce two benchmarks for instruction following: LANI, a navigation task; and CHAI, where an agent executes household instructions. Our evaluation demonstrates the advantages of our model decomposition, and illustrates the challenges posed by our new benchmarks.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

clic-lab/ciff officialmentioned in papermentioned on GitHubpytorchGPL-3.0Syntology: not harvested report
clic-lab/chalet mentioned on GitHubSyntology: not harvested report
clic-lab/drif mentioned on GitHubpytorchSyntology: not harvested report
lil-lab/chalet mentioned on GitHubSyntology: not harvested report
lil-lab/ciff mentioned on GitHubpytorchGPL-3.0Syntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action GenerationConditional Image GenerationImage GenerationInstruction Following

Datasets

Introduced by this paper, per the archive.

Lani

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections