Authors
- W. Bradley Knox*
- Sigurdur Orn Adalgeirsson*
- Serena Booth*
- Anca Dragan*
- Peter Stone
- Scott Niekum*
* External authors
Venue
- AAAI-24
Date
- 2024
Share
Learning Optimal Advantage from Preferences and Mistaking it for Reward.
W. Bradley Knox*
Sigurdur Orn Adalgeirsson*
Serena Booth*
Anca Dragan*
Scott Niekum*
* External authors
AAAI-24
2024
Abstract
We consider algorithms for learning reward functions from human preferences over pairs of trajectory segments, as used in reinforcement learning from human feedback (RLHF). Most recent work assumes that human preferences are generated based only upon the reward accrued within those segments, or their partial return. Recent work casts doubt on the validity of this assumption, proposing an alternative preference model based upon regret. We investigate the consequences of assuming preferences are based upon partial return when they actually arise from regret. We argue that the learned function is an approximation of the optimal advantage function,
Related Publications
Having explored an environment, intelligent agents should be able to transfer their knowledge to most downstream tasks within that environment. Referred to as ``zero-shot learning," this ability remains elusive for general-purpose reinforcement learning algorithms. While rec…
Scaling up the model size and computation has brought consistent performance improvements in supervised learning. However, this lesson often fails to apply to reinforcement learning (RL) because training the model on non-stationary data easily leads to overfitting and unstab…
Deep reinforcement learning has achieved superhuman racing performance in high-fidelity simulators like Gran Turismo 7 (GT7). It typically utilizes global features that require instrumentation external to a car, such as precise localization of agents and opponents, limiting …
JOIN US
Shape the Future of AI with Sony AI
We want to hear from those of you who have a strong desire
to shape the future of AI.
