Event Tables for Efficient Experience Replay

Varun Kompella

Thomas Walsh

Samuel Barrett

Peter R. Wurman

Peter Stone

CoLLAs 2023



Experience replay (ER) is a crucial component of many deep reinforcement learning (RL) systems. However, uniform sampling from an ER buffer can lead to slow convergence and unstable asymptotic behaviors. This paper introduces Stratified Sampling from Event Tables (SSET), which partitions an ER buffer into Event Tables, each capturing important subsequences of optimal behavior. We prove a theoretical advantage over the traditional monolithic buffer approach and combine SSET with an existing prioritized sampling strategy to further improve learning speed and stability. Empirical results in challenging MiniGrid domains, benchmark RL environments, and a high-fidelity car racing simulator demonstrate the advantages and versatility of SSET over existing ER buffer sampling approaches.

Related Publications

Symbolic State Space Optimization for Long Horizon Mobile Manipulation Planning.

International Conference on Intelligent Robots and Systems, 2023
Xiaohan Zhang*, Yifeng Zhu*, Yan Ding*, Yuqian Jiang*, Yuke Zhu*, Peter Stone, Shiqi Zhang*

In existing task and motion planning (TAMP) research, it is a common assumption that experts manually specify the state space for task-level planning. A welldeveloped state space enables the desirable distribution of limited computational resources between task planning an…

Composing Efficient, Robust Tests for Policy Selection

UAI, 2023
Dustin Morrill, Thomas Walsh, Daniel Hernandez, Peter R. Wurman, Peter Stone

Modern reinforcement learning systems produce many high-quality policies throughout the learning process. However, to choose which policy to actually deploy in the real world, they must be tested under an intractable number of environmental conditions. We introduce RPOSST, a…

Model-Based Meta Automatic Curriculum Learning.

CoLLAs, 2023
Zifan Xu*, Yulin Zhang*, Shahaf S. Shperberg*, Reuth Mirsky*, Yuqian Jiang*, Bo Liu*, Peter Stone

Curriculum learning (CL) has been widely explored to facilitate the learning of hard-exploration tasks in reinforcement learning (RL) by training a sequence of easier tasks, often called a curriculum. While most curricula are built either manually or automatically based on h…


Shape the Future of AI with Sony AI

We want to hear from those of you who have a strong desire
to shape the future of AI.