عنوان انگلیسی مقاله:
A model-based deep reinforcement learning method applied to finite-horizon optimal control of nonlinear control-affine system
ترجمه فارسی عنوان مقاله:
یک روش یادگیری تقویتی عمیق مبتنی بر مدل اعمال شده برای کنترل بهینه افق محدود سیستم کنترل غیر خطی
Sciencedirect - Elsevier - Journal of Process Control, 87 (2020) 166-178. doi:10.1016/j.jprocont.2020.02.003
Jong Woo Kim a , Byung Jun Park a , Haeun Yoo b , Tae Hoon Oh a , Jay H. Lee b , ∗, Jong Min Lee a
The Hamilton–Jacobi–Bellman (HJB) equation can be solved to obtain optimal closed-loop control policies for general nonlinear systems. As it is seldom possible to solve the HJB equation exactly for nonlinear systems, either analytically or numerically, methods to build approximate solutions through simulation based learning have been studied in various names like neurodynamic programming (NDP) and approx- imate dynamic programming (ADP). The aspect of learning connects these methods to reinforcement learning (RL), which also tries to learn optimal decision policies through trial-and-error based learning. This study develops a model-based RL method, which iteratively learns the solution to the HJB and its associated equations. We focus particularly on the control-affine system with a quadratic objective func- tion and the finite horizon optimal control (FHOC) problem with time-varying reference trajectories. The HJB solutions for such systems involve time-varying value, costate, and policy functions subject to bound- ary conditions. To represent the time-varying HJB solution in high-dimensional state space in a general and efficient way, deep neural networks (DNNs) are employed. It is shown that the use of DNNs, com- pared to shallow neural networks (SNNs), can significantly improve the performance of a learned policy in the presence of uncertain initial state and state noise. Examples involving a batch chemical reactor and a one-dimensional diffusion-convection-reaction system are used to demonstrate this and other key aspects of the method.
Keywords: Reinforcement learning | Approximate dynamic programming | Deep neural networks | Globalized dual heuristic programming | Finite horizon optimal control problem | Hamilton–Jacobi–Bellman equation