Composing Efficient, Robust Tests for Policy Selection

Dustin Morrill

Thomas Walsh

Daniel Hernandez

Peter R. Wurman

Peter Stone

UAI 2023



Modern reinforcement learning systems produce many high-quality policies throughout the learning process. However, to choose which policy to actually deploy in the real world, they must be tested under an intractable number of environmental conditions. We introduce RPOSST, an algorithm to select a small set of test cases from a larger pool based on a relatively small number of sample evaluations. RPOSST treats the test case selection problem as a two-player game and optimizes a solution with provable k-of-N robustness, bounding the error relative to a test that used all the test cases in the pool. Empirical results demonstrate that RPOSST finds a small set of test cases that identify high quality policies in a toy one-shot game, poker datasets, and a high-fidelity racing simulator.

Related Publications

Symbolic State Space Optimization for Long Horizon Mobile Manipulation Planning.

International Conference on Intelligent Robots and Systems, 2023
Xiaohan Zhang*, Yifeng Zhu*, Yan Ding*, Yuqian Jiang*, Yuke Zhu*, Peter Stone, Shiqi Zhang*

In existing task and motion planning (TAMP) research, it is a common assumption that experts manually specify the state space for task-level planning. A welldeveloped state space enables the desirable distribution of limited computational resources between task planning an…

Event Tables for Efficient Experience Replay

CoLLAs, 2023
Varun Kompella, Thomas Walsh, Samuel Barrett, Peter R. Wurman, Peter Stone

Experience replay (ER) is a crucial component of many deep reinforcement learning (RL) systems. However, uniform sampling from an ER buffer can lead to slow convergence and unstable asymptotic behaviors. This paper introduces Stratified Sampling from Event Tables (SSET), whi…

BRExIt: On Opponent Modelling in Expert Iteration

IJCAI, 2023
Daniel Hernandez, Hendrik Baier*, Michael Kaisers*

Finding a best response policy is a central objective in game theory and multi-agent learning, with modern population-based training approaches employing reinforcement learning algorithms as best response oracles to improve play against candidate opponents (typically previou…


Shape the Future of AI with Sony AI

We want to hear from those of you who have a strong desire
to shape the future of AI.