Residual-MPPI: Online Policy Customization for Continuous Control

Pengcheng Wang

Chenran Li

Catherine Weaver*

Kenta Kawamoto

Masayoshi Tomizuka*

Chen Tang*

Wei Zhan*

* External authors

ICLR-25

2025

Abstract

Policies learned through Reinforcement Learning (RL) and Imitation
Learning (IL) have demonstrated significant potential in achieving advanced performance in continuous control tasks. However, in real-world environments, it
is often necessary to further customize a trained policy when there are additional requirements that were unforeseen during the original training phase. It
is possible to fine-tune the policy to meet the new requirements, but this often requires collecting new data with the added requirements and access to the
original training metric and policy parameters. In contrast, an online planning
algorithm, if capable of meeting the additional requirements, can eliminate the
necessity for extensive training phases and customize the policy without knowledge of the original training scheme or task. In this work, we propose a generic
online planning algorithm for customizing continuous-control policies at the execution time which we call Residual-MPPI. It is able to customize a given prior
policy on new performance metrics in few-shot and even zero-shot online settings. Also, Residual-MPPI only requires access to the action distribution produced by the prior policy, without additional knowledge regarding the original
task. Through our experiments, we demonstrate that the proposed ResidualMPPI algorithm can accomplish the few-shot/zero-shot online policy customization task effectively, including customizing the champion-level racing agent, Gran
Turismo Sophy (GT Sophy) 1.0, in the challenging car racing scenario, Gran
Turismo Sport (GTS) environment. Demo videos are available on our website:
https://sites.google.com/view/residual-mppi