با سلام خدمت کاربران در صورتی که با خطای سیستم پرداخت بانکی مواجه شدید از طریق کارت به کارت (6037997535328901 بانک ملی ناصر خنجری ) مقاله خود را دریافت کنید (تا مشکل رفع گردد).
دسته بندی:
داده های بزرگ - big data
سال انتشار:
2018
عنوان انگلیسی مقاله:
Learning-Based Memory Allocation Optimization for Delay-Sensitive Big Data Processing
ترجمه فارسی عنوان مقاله:
بهینه سازی اختصاص حافظه بر مبنای یادگیری برای پردازش داده های بزرگ با حساسیت تاخیری
منبع:
IEEE - IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL: 29, NO: 6, JUNE 2018
نویسنده:
Linjiun Tsai, Hubertus Franke, Chung-Sheng L،Wanjiun Liao
چکیده انگلیسی:
Optimal resource provisioning is essential for scalable big data analytics. However, it has been difficult to accurately
forecast the resource requirements before the actual deployment of these applications as their resource requirements are heavily
application and data dependent. This paper identifies the existence of effective memory resource requirements for most of the big data
analytic applications running inside JVMs in distributed Spark environments. Provisioning memory less than the effective memory
requirement may result in rapid deterioration of the application execution in terms of its total execution time. A machine learning-based
prediction model is proposed in this paper to forecast the effective memory requirement of an application given its service level
agreement. This model captures the memory consumption behavior of big data applications and the dynamics of memory utilization in
a distributed cluster environment. With an accurate prediction of the effective memory requirement, it is shown that up to 60 percent
savings of the memory resource is feasible if an execution time penalty of 10 percent is acceptable. The accuracy of the model is
evaluated on a physical Spark cluster with 128 cores and 1TB of total memory. The experiment results show that the proposed solution
can predict the minimum required memory size for given acceptable delays with high accuracy, even if the behavior of target
applications is unknown during the training of the model.
Index Terms: Big data, spark, memory over-commitment, garbage collection, profiling, modeling, performance-cost tradeoff
قیمت: رایگان
توضیحات اضافی:
تعداد نظرات : 0