عنوان انگلیسی مقاله:
Cost-efficient dynamic scheduling of big data applications in apache spark on cloud
ترجمه فارسی عنوان مقاله:
برنامه ریزی پویا مقرون به صرفه برنامه های داده های بزرگ در آپاچی اسپارک روی ابر
Sciencedirect - Elsevier - The Journal of Systems & Software, 162 (2020) : doi:10:1016/j:jss:2019:110515
Muhammed Tawfiqul Islam a , ∗, Satish Narayana Sriramaa , b , Shanika Karunasekeraa , Rajkumar Buyya a
Job scheduling is one of the most crucial components in managing resources, and efficient execution of big data applications. Specifically, scheduling jobs in a cloud-deployed cluster are challenging as the cloud offers different types of Virtual Machines (VMs) and jobs can be heterogeneous. The default big data processing framework schedulers fail to reduce the cost of VM usages in the cloud environment while satisfying the performance constraints of each job. The existing works in cluster scheduling mainly focus on improving job performance and do not leverage from VM types on the cloud to reduce cost. In this paper, we propose efficient scheduling algorithms that reduce the cost of resource usage in a cloud-deployed Apache Spark cluster. In addition, the proposed algorithms can also prioritise jobs based on their given deadlines. Besides, the proposed scheduling algorithms are online and adaptive to clus- ter changes. We have also implemented the proposed algorithms on top of Apache Mesos. Furthermore, we have performed extensive experiments on real datasets and compared to the existing schedulers to showcase the superiority of our proposed algorithms. The results indicate that our algorithms can reduce resource usage cost up to 34% under different workloads and improve job performance.
Keywords: Cloud | Apache spark | Scheduling | Cost-efficiency