Papers › Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings

Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings

4 Mar 2021NeurIPS 2021 12arXiv:2103.02886archive 2025-07-28

Lili Chen, Kimin Lee, Aravind Srinivas, Pieter Abbeel

Recent advances in off-policy deep reinforcement learning (RL) have led to impressive success in complex tasks from visual observations. Experience replay improves sample-efficiency by reusing experiences from the past, and convolutional neural networks (CNNs) process high-dimensional inputs effectively. However, such techniques demand high memory and computational bandwidth. In this paper, we present Stored Embeddings for Efficient Reinforcement Learning (SEER), a simple modification of existing off-policy RL methods, to address these computational and memory requirements. To reduce the computational overhead of gradient updates in CNNs, we freeze the lower layers of CNN encoders early in training due to early convergence of their parameters. Additionally, we reduce memory requirements by storing the low-dimensional latent vectors for experience replay instead of high-dimensional images, enabling an adaptive increase in the replay buffer capacity, a useful technique in constrained-memory settings. In our experiments, we show that SEER does not degrade the performance of RL agents while significantly saving computation and memory across a diverse set of DeepMind Control environments and Atari games.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

lili-chen/SEER officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Atari Games Computational Efficiency Deep Reinforcement Learning Reinforcement Learning Reinforcement Learning (RL)Transfer Learning reinforcement-learning

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Atari Games	Atari 2600 Alien	Rainbow+SEER	Score	1172.6	#36 of 49	Archive leaderboard	report
Atari Games	Atari 2600 Amidar	Rainbow+SEER	Score	250.5	#33 of 48	Archive leaderboard	report
Atari Games	Atari 2600 Bank Heist	Rainbow+SEER	Score	276.6	#39 of 45	Archive leaderboard	report
Atari Games	Atari 2600 Crazy Climber	Rainbow+SEER	Score	28066	#43 of 49	Archive leaderboard	report
Atari Games	Atari 2600 Krull	Rainbow+SEER	Score	3277.5	#45 of 45	Archive leaderboard	report
Atari Games	Atari 2600 Q*Bert	Qbert Rainbow+SEER	Score	4123.5	#45 of 57	Archive leaderboard	report
Atari Games	Atari 2600 Road Runner	Rainbow+SEER	Score	11794	#39 of 44	Archive leaderboard	report
Atari Games	Atari 2600 Seaquest	Rainbow+SEER	Score	561.2	#52 of 57	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

1x1 Convolution Average Pooling Batch Normalization Convolution Cosine Annealing Dense Connections Experience Replay Global Average Pooling Grouped Convolution LARS ReLU RegNetY SEER Sigmoid Activation Squeeze-and-Excitation Block SwAV

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections