عنوان انگلیسی مقاله:
PMA-DRL: A parallel model-augmented framework for deep reinforcement learning algorithms
ترجمه فارسی عنوان مقاله:
PMA-DRL: چارچوبی با مدل تقویت شده موازی برای الگوریتم های یادگیری تقویتی عمیق
Sciencedirect - Elsevier - Neurocomputing 403 (2020) 109–120
Xufang Luo a, Yunhong Wanga,⇑
In recent years, aided by deep neural networks (DNNs), reinforcement learning (RL) algorithms have been
achieving great success in more and more tasks. In general, model-free RL algorithms are widely applicable,
but sometimes suffer from low sample efficiency. Although the environment dynamics can be
incorporated into model-free RL algorithms to enhance sample efficiency, which can be transformed into
the joint algorithm of model-free and model-based RL algorithms, the model bias of dynamics model may
still hurt the performance. Another attractive study direction for RL algorithms, is using parallel strategy.
Meanwhile, parallel RL algorithms can achieve outstanding results under less time cost, but are still with
low sample efficiency, because most of such algorithms are vanilla model-free ones. Therefore, aiming to
enhance the model performance for RL algorithms with DNNs, in this paper, we propose a novel parallel
model-augmented framework, called PMA-DRL, to combine the dynamics model and parallel RL algorithms
together. First, we introduce a local dynamics model (LDM) for each local agent model (LAM) in
parallel RL algorithms. Next, we propose to build a better LDM by using an ensemble of LDMs, and such
ensemble can help to reduce model bias, because the experienced observations under different LAMs are
always diverse. Then, the observation diversity can be further increased by employing LDMs to explore.
Finally, we implement experiments on classic control tasks (CCTs) and Atari games, which demonstrates
the efficiency of our proposed framework.
Keywords: Reinforcement learning | Environment dynamics | Parallel framework