A new generation of smart stormwater systems promises to reduce the need for new construction by enhancing the performance of the existing infrastructure through real-time control. Smart stormwater systems dynamically adapt their response to individual storms by controlling distributed assets, such as valves, gates, and pumps. This paper introduces a real-time control approach based on Reinforcement Learning (RL), which has emerged as a state-of-the-art methodology for autonomous control in the artificial intelligence community. Using a Deep Neu- ral Network, a RL-based controller learns a control strategy by interacting with the system it controls - effectively trying various control strategies until converging on those that achieve a desired objective. This paper formulates and implements a RL algorithm for the real-time control of urban stormwater systems. This algorithm trains a RL agent to control valves in a distributed stormwater system across thousands of simulated storm scenarios, seeking to achieve water level and flow set-points in the system. The algorithm is first evaluated for the control of an individual stormwater basin, after which it is adapted to the control of multiple basins in a larger watershed (4 km 2 ). The results indicate that RL can very effectively control individual sites. Performance is highly sensitive to the reward formulation of the RL agent. Generally, more explicit guidance led to better control performance, and more rapid and stable convergence of the learning process. While the control of multiple distributed sites also shows promise in reducing flooding and peak flows, the complexity of controlling larger systems comes with a number of caveats. The RL controller’s performance is very sensitive to the formulation of the Deep Neural Network and requires a significant amount of computational resource to achieve a reasonable performance en- hancement. Overall, the controlled system significantly outperforms the uncontrolled system, especially across storms of high intensity and duration. A frank discussion is provided, which should allow the benefits and draw- backs of RL to be considered when implementing it for the real-time control of stormwater systems. An open source implementation of the full simulation environment and control algorithms is also provided.
Keywords: Real-time control | Reinforcement learning | Smart stormwater systems

The optimization and training processes of deep reinforcement learning (DRL) based energy management strategy (EMS) can be very slow and resource-intensive. In this paper, an improved energy management framework that embeds expert knowledge into deep deterministic policy gradient (DDPG) is proposed. Incorporated with the battery characteristics and the optimal brake specific fuel consumption (BSFC) curve of hybrid electric vehicles (HEVs), we are committed to solving the optimization problem of multi-objective energy management with a large space of control variables. By incorporating this prior knowledge, the proposed framework not only accelerates the learning process, but also gets a better fuel economy, thus making the energy management system relatively stable. The experimental results show that the proposed EMS outperforms the one without prior knowledge and the other state-of-art deep reinforcement learning approaches. In addition, the proposed approach can be easily generalized to other types of HEV EMSs.
Keywords: Energy management strategy | Hybrid electric vehicle | Expert knowledge | Deep deterministic policy gradient | Continuous action space

Wind speed forecasting is a promising solution to improve the efficiency of energy utilization. In this study, a novel hybrid wind speed forecasting model is proposed. The whole modeling process of the proposed model consists of three steps. In stage I, the empirical wavelet transform method reduces the non-stationarity of the original wind speed data by decomposing the original data into several subseries. In stage II, three kinds of deep networks are utilized to build the forecasting model and calculate prediction results of all sub-series, respectively. In stage III, the reinforcement learning method is used to combine three kinds of deep networks. The forecasting results of each sub-series are combined to obtain the final forecasting results. By comparing all the results of the predictions over three different types of wind speed series, it can be concluded that: (a) the proposed reinforcement learning based ensemble method is effective in integrating three kinds of deep network and works better than traditional optimization based ensemble method; (b) the proposed ensemble deep reinforcement learning based wind speed prediction model can get accurate results in all cases and provide the best accuracy compared with sixteen alternative models and three state-of-the-art models.
Keywords: Wind speed forecasting | Ensemble deep reinforcement learning | Empirical wavelet transform | Hybrid wind speed forecasting model

In this paper, a truck scheduling problem at a cross-docking center is investigated where inbound trucks are also used as outbound. Moreover, inbound trucks do not need to unload and reload the demand of allocated destination, i.e. they can be partially unloaded. The problem is modeled as a mixed integer program to find the optimal dock-door and destination assignments as well as the scheduling of trucks to minimize makespan. Due to model complexity, a hybrid heuristic-simulated annealing is developed. A number of generic and tailor-made neighborhood search structures are also developed to efficiently search solution space. Moreover, some reinforcement learning methods are applied to intellectually learn more suitable neighborhood search structures in different situations. Finally, the numerical study shows that partial unloading of compound trucks has a crucial impact on makespan reduction.
Keywords: Logistics | Cross docking | Truck scheduling | Simulated annealing | Reinforcement learning

Driven by the recent advances in industry 4.0 and industrial artificial intelligence, Automated Guided Vehicles (AGVs) has been widely used in flexible shop floor for material handling. However, great challenges aroused by the high dynamics, complexity, and uncertainty of the shop floor environment still exists on AGVs real-time scheduling. To address these challenges, an adaptive deep reinforcement learning (DRL) based AGVs real-time scheduling approach with mixed rule is proposed to the flexible shop floor to minimize the makespan and delay ratio. Firstly, the problem of AGVs real-time scheduling is formulated as a Markov Decision Process (MDP) in which state representation, action representation, reward function, and optimal mixed rule policy, are described in detail. Then a novel deep q-network (DQN) method is further developed to achieve the optimal mixed rule policy with which the suitable dispatching rules and AGVs can be selected to execute the scheduling towards various states. Finally, the case study based on a real-world flexible shop floor is illustrated and the results validate the feasibility and effectiveness of the proposed approach.
Keywords: Automated guided vehicles | Real-time scheduling | Deep reinforcement learning | Industry 4.0