Is RL fine-tuning harder than regression? A PDE learning approach for diffusion models

arXiv:2509.02528v1 Announce Type: cross Abstract: We study the problem of learning the optimal control policy for fine-tuning a given diffusion process, using general value function approximation. We develop a new class of algorithms by solving a variational inequality problem based on the Hamilton-Jacobi-Bellman (HJB) equations. We prove sharp statistical rates for the learned value function and control policy, depending on the complexity and approximation errors of the function class. In contrast to generic reinforcement learning problems, our approach shows that fine-tuning can be achieved via supervised regression, with faster statistical rate guarantees.

2025-09-03 04:00 GMT · 1 day ago arxiv.org
2025-09-03 04:00 GMT · 1 day ago arxiv.org

arXiv:2509.02528v1 Announce Type: cross Abstract: We study the problem of learning the optimal control policy for fine-tuning a given diffusion process, using general value function approximation. We develop a new class of algorithms by solving a variational inequality problem based on the Hamilton-Jacobi-Bellman (HJB) equations. We prove sharp statistical rates for the learned value function and control policy, depending on the complexity and approximation errors of the function class. In contrast to generic reinforcement learning problems, our approach shows that fine-tuning can be achieved via supervised regression, with faster statistical rate guarantees.

Original: https://arxiv.org/abs/2509.02528