Machine Learning and Dynamical Systems (MLDS 5) - Conference
Type
conference lecture
Date Issued
2026-02-10
Author(s)
Abstract
We study offline reinforcement learning problems with a long-run average reward objective. The
state-action pairs generated by any fixed behavioral policy thus follow a Markov chain, and the
empirical state-action-next-state distribution satisfies a large deviations principle. We use the rate
function of this large deviations principle to construct an uncertainty set for the unknown true
state-action-next-state distribution. We also construct a distribution shift transformation that maps
any distribution in this uncertainty set to a state-action-next-state distribution of the Markov chain
generated by a fixed evaluation policy, which may differ from the unknown behavioral policy. We
prove that the worst-case average reward of the evaluation policy with respect to all distributions in
the shifted uncertainty set provides, in a rigorous statistical sense, the least conservative estimator
for the average reward under the unknown true distribution. This guarantee is available even if one
has only access to one single trajectory of serially correlated state-action pairs. The emerging robust
optimization problem can be viewed as a robust Markov decision process with a non-rectangular
uncertainty set. We adapt an efficient policy gradient algorithm to solve this problem. Numerical
experiments show that our methods compare favorably against state-of-the-art methods.
state-action pairs generated by any fixed behavioral policy thus follow a Markov chain, and the
empirical state-action-next-state distribution satisfies a large deviations principle. We use the rate
function of this large deviations principle to construct an uncertainty set for the unknown true
state-action-next-state distribution. We also construct a distribution shift transformation that maps
any distribution in this uncertainty set to a state-action-next-state distribution of the Markov chain
generated by a fixed evaluation policy, which may differ from the unknown behavioral policy. We
prove that the worst-case average reward of the evaluation policy with respect to all distributions in
the shifted uncertainty set provides, in a rigorous statistical sense, the least conservative estimator
for the average reward under the unknown true distribution. This guarantee is available even if one
has only access to one single trajectory of serially correlated state-action pairs. The emerging robust
optimization problem can be viewed as a robust Markov decision process with a non-rectangular
uncertainty set. We adapt an efficient policy gradient algorithm to solve this problem. Numerical
experiments show that our methods compare favorably against state-of-the-art methods.
Language
English
Keywords
Offline Reinforcement Learning
Off-Policy Evaluation
Large Deviations Theory
Markov Decision Processes
Distributionally Robust Optimization
HSG Classification
contribution to scientific community
Refereed
No
Event Location
Kyoto University
Event Date
February 9-13, 2026
Subject(s)
Division(s)
Additional Information
https://sites.google.com/view/mlds-deds-2026/%E3%83%9B%E3%83%BC%E3%83%A0?authuser=0
File(s)![Thumbnail Image]()
Name
Optimal_Offline_RL.pdf
Size
837.01 KB
Format
Adobe PDF
Checksum (MD5)
2bd95d937d1fdbd064bea4a99c925144