Optimal policy for structure maintenance: A deep reinforcement learning framework
سیاست بهینه برای نگهداری ساختار: یک چارچوب یادگیری تقویت عمیق-2020
The cost-effective management of aged infrastructure is an issue of worldwide concern. Markov decision process (MDP) models have been used in developing structural maintenance policies. Recent advances in the artificial intelligence (AI) community have shown that deep reinforcement learning (DRL) has the potential to solve large MDP optimization tasks. This paper proposes a novel automated DRL framework to obtain an optimized structural maintenance policy. The DRL framework contains a decision maker (AI agent) and the structure that needs to be maintained (AI task environment). The agent outputs maintenance policies and chooses maintenance actions, and the task environment determines the state transition of the structure and returns rewards to the agent under given maintenance actions. The advantages of the DRL framework include: (1) a deep neural network (DNN) is employed to learn the state-action Q value (defined as the predicted discounted expectation of the return for consequences under a given state-action pair), either based on simulations or historical data, and the policy is then obtained from the Q value; (2) optimization of the learning process is sample-based so that it can learn directly from real historical data collected from multiple bridges (i.e., big data from a large number of bridges); and (3) a general framework is used for different structure maintenance tasks with minimal changes to the neural network architecture. Case studies for a simple bridge deck with seven components and a long-span cable-stayed bridge with 263 components are performed to demonstrate the proposed procedure. The results show that the DRL is efficient at finding the optimal policy for maintenance tasks for both simple and complex structures.
Keywords: Bridge maintenance policy | Deep reinforcement learning (DRL) | Markov decision process (MDP) | Deep Q-network (DQN) | Convolutional neural network (CNN)
Intelligent fault diagnosis for rotating machinery using deep Q-network based health state classification: A deep reinforcement learning approach
تشخیص خطای هوشمند برای ماشین آلات در حال چرخش با استفاده از طبقه بندی حالت سلامت مبتنی بر شبکه Q عمقی: یک روش یادگیری تقویتی عمیق-2019
Fault diagnosis methods for rotating machinery have always been a hot research topic, and artificial intelligencebased approaches have attracted increasing attention from both researchers and engineers. Among those related studies and methods, artificial neural networks, especially deep learning-based methods, are widely used to extract fault features or classify fault features obtained by other signal processing techniques. Although such methods could solve the fault diagnosis problems of rotating machinery, there are still two deficiencies. (1) Unable to establish direct linear or non-linear mapping between raw data and the corresponding fault modes, the performance of such fault diagnosis methods highly depends on the quality of the extracted features. (2) The optimization of neural network architecture and parameters, especially for deep neural networks, requires considerable manual modification and expert experience, which limits the applicability and generalization of such methods. As a remarkable breakthrough in artificial intelligence, AlphaGo, a representative achievement of deep reinforcement learning, provides inspiration and direction for the aforementioned shortcomings. Combining the advantages of deep learning and reinforcement learning, deep reinforcement learning is able to build an end-to-end fault diagnosis architecture that can directly map raw fault data to the corresponding fault modes. Thus, based on deep reinforcement learning, a novel intelligent diagnosis method is proposed that is able to overcome the shortcomings of the aforementioned diagnosis methods. Validation tests of the proposed method are carried out using datasets of two types of rotating machinery, rolling bearings and hydraulic pumps, which contain a large number of measured raw vibration signals under different health states and working conditions. The diagnosis results show that the proposed method is able to obtain intelligent fault diagnosis agents that can mine the relationships between the raw vibration signals and fault modes autonomously and effectively. Considering that the learning process of the proposed method depends only on the replayed memories of the agent and the overall rewards, which represent much weaker feedback than that obtained by the supervised learning-based method, the proposed method is promising in establishing a general fault diagnosis architecture for rotating machinery.
Keywords: Fault diagnosis | Rotating machinery | Deep reinforcement learning | Deep Q-network
Designing online network intrusion detection using deep auto-encoder Q-learning
طراحی تشخیص نفوذ آنلاین به شبکه با استفاده از یادگیری-Q خودرمزگذار عمیق-2019
Because of the increasing application of reinforcement learning (RL), particularly deep Q- learning algorithm, research organizations utilize it with increasing frequency. The predic- tion of cyber vulnerability and development of efficient real-time online network intrusion detection (NID) systems are progressions toward becoming RL-powered. An open issues in NID is the model design and prediction of real-time online data composed of a series of time-related feature patterns. There have been concerns regarding the operation of the developed systems because cyber-attack scenarios vary continuously to circumvent NID. These issues have been related to the human interaction significance and the decrease in accuracy verification. Therefore, we employ an RL that permits a deep auto-encoder in the Q-network (DAEQ-N). The proposed DAEQ-N attempts to achieve the maximum prediction accuracy in online learning systems into which continuous behavior patterns are fed and which are trained with more significant weights by classifying it as either “normal”or “anomalous.”
Keywords: Network anomalies | Online learning systems | Network intrusion detection (NID) | Deep Q-Network (DQN) | Reinforcement learning (RL)