报告预览

5.李呈祥 -Apache Spark最新技术进展和3.0+展望（21页）.pdf

编号：83829

PDF 21页 2.02MB 下载积分：VIP专享

下载报告请您先登录！

5.李呈祥 -Apache Spark最新技术进展和3.0+展望（21页）.pdf

1、Apache Spark最新技术进展和3.0+展望李呈祥（司麟）阿里巴巴高级技术专家计算平台事业部Agenda3.0Spark on CloudData Warehouse EnhancementSpark+AIA Unified Analytics Engine for Large-scale Data ProcessingEasy-to-use APIRich Ecosystem SupportEfficient EngineData Warehouse EnhancementDelta Lake ACID Transactions Scalable Metadata Handling T

2、ime Travel(data versioning)Open Format Unified Batch and Streaming Source and Sink Schema EnforcementComing soon:Audit HistoryFull DML SupportExpectationsData Source V2 Unified API for batch and streaming Flexible API for high performance implementation Flexible API for metadata management Target 3.

3、0Runtime OptimizationDynamic optimize the execution plan at runtime based on the statistic of previous stage.Self tuning the number of reducers Adaptive join strategy Automatic skew join handlingAdaptive ExecutionEMR Runtime Filter Filter big table with runtime statistic of join key.Support both par

4、titioned table and normal table.EMR Spark Relational CacheUser may analyze data in certain access patternRegularly join 2 tables?Regularly aggregate by certain fields?Regularly filter by certain fields?Data Organization:partition,bucket,sortfile index,zorderData pre-computation:pre-filterdenormaliza

5、tionpre-aggregationMake data adaptive to compute,so spark compute faster.EMR Spark Relational CacheEMR Spark Relational CacheEasy to build and maintainTransparent to userCREATE VIEW emp_flat AS SELECT*FROM employee,address WHERE e_addrId=a_addrId;CACHE TABLE emp_flatUSING parquetPARTITIONED BY(e_ob_

6、date)EAJFP-User Query-SELECT*FROM employee,address WHERE e_addrId=a_addrId and a_cityName=ShangHaiSpark OptimizerCFPEAJP-Cached Mata-emp_flatoptimized planSpark on CloudStorage and Computing DisaggregationWhy disaggregate storage and computing:Pay as you go.Scale independently of each other.More rel

7、iable storage.Aliyun OSSThe challenge of disaggregation:Object store metadata management.Limited network resource.Storage and Computing DisaggregationEMR JindoFS fill the gap between object store and compute framework:File System API and meta management.Local replication support.Remote reliable stor

8、age and fast local access.Automatic and transparent cold data separation and migration Optimized for machine learning and Spark AISpark on Cloud:Remote Shuffle ServiceData source storage is disaggregated from computing while local shuffle data is not.Local storage has poor elasticity.Current externa

9、l shuffle service make cost extra effort for worker/nodemanager,and is not available for k8s.SPARK-25299 would support write shuffle file to remote storage,remote shuffle service is still on the way.Remote Shuffle ServicePay as you go.Service run on hosts optimized for it.Spark ClusterMRMRMRSpark Cl

10、usterMRMRMRSpark on KubernetesNatively support since 2.3Pyspark/R binding and client mode supported since 2.4Spark 3.0+Dynamic allocation supportKerberos supportSpark+AIProject Hydrogen:Spark+AIBarrier Execution ModeAccelerator Aware SchedulingOptimized Data ExchangeBetter AI need big dataData analy

11、sis get deeperHydrogen make Spark a unified AI processing pipelineProject Hydrogen:Barrier Execution Gang scheduling enabled to run DL job as Spark stage.Specific recovery strategy supported for gang scheduled stage.Available since 2.4Task1Task2Task3Task1Task2Task3SparkMLProject Hydrogen:Accelerator

12、 Aware SchedulingGPUs are applied at application level.User can retrieve assigned GPUs from task context.Can extend to other accelerator,such as:FPGAAvailable at 3.0,see SPARK-27362,SPARK-27363Project Hydrogen:Optimized Data ExchangedataSpark loads/saves data from/to persistent storage in a data for

13、mat used by a DL/AI framework.Spark feeds data into DL/AI frameworks for training.Prefer to use Apache Arrow as exchange data format.SPARK-24615 WIPSpark 3.03.0 TargetsProject HydrogenGPU-Aware schedulingOptimized data exchangeAdaptive ExecutionSelf tuning the number of reducersAdaptive join strateg

14、yData Source V2Spark on K8sDynamic resource allocationKerberos supportHadoop 3.x supportHive 2.3 supportScala 2.12 GABetter ANSI SQL complianceThis presentation may contain projections or other forward-looking statements regarding the upcoming release(Apache Spark 3.0).The statements are intended to outline our general direction.They are intended for information purposes only.They are not a commitment to deliver code or functionality.The development,release and timing of any feature or functionality described for Apache Spark remains at the sole discretion of ASF and the Apache Spark PMC.

友情提示

1、下载报告失败解决办法
2、PDF文件下载后，可能会被浏览器默认打开，此种情况可以点击浏览器菜单，保存网页到桌面，就可以正常下载了。
3、本站不支持迅雷下载，请使用电脑自带的IE浏览器，或者360浏览器、谷歌浏览器下载即可。
4、本站报告下载后的文档和图纸-无水印,预览文档经过压缩，下载后原文更清晰。

本文（5.李呈祥 -Apache Spark最新技术进展和3.0+展望（21页）.pdf）为本站（小时候）主动上传，三个皮匠报告文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若此文所含内容侵犯了您的版权或隐私，请立即通知三个皮匠报告文库（点击联系客服），我们立即给予删除！

温馨提示：如果因为网速或其他原因下载失败请重新下载，重复下载不扣分。

上海品茶

5.李呈祥 -Apache Spark最新技术进展和3.0+展望（21页）.pdf

5.李呈祥 -Apache Spark最新技术进展和3.0+展望（21页）.pdf