1、S P A R K +V O L C A N O:S P A R K +V O L C A N O:为为 S p a r k o n K u b e r n e t e sS p a r k o n K u b e r n e t e s 提 供提 供更 完 善 的 调 度 能 力！更 完 善 的 调 度 能 力！姜逸坤 华为计算开源团队|从一个简单而又悲伤的数学题说起：已知：已知：-用于Spark的Kubernetes集群CPU总量：2.02.0-每个Job所需CPU总量：2.02.0，完成单个Job所需时间为：1010秒秒问题：问题：请问，并发提交3 3个个JobJob，多久可以完成？实际

2、上的你以为的30s喵之自信0101Spark on K8SSpark on K8S现状与发展现状与发展0202Spark 3.3Spark 3.3自定义调度支持自定义调度支持0303Spark Spark VolcanoVolcano设计设计与实现与实现0404DemoDemo演示演示 目录目录 CONTENT|Spark on K8SSpark on K8S现状与发展现状与发展01|Apache Spark社区贡献趋势2016-2022年 Apache Spark 资源管理领域代码提交趋势 12016-2022年 Apache Spark 资源管理领域的搜索趋势 21 数据来自于Spark合

3、入Commits，通过Git过滤标题包含YARNK8SMESOS的个数统计。2 数据来自于Google Trend。Spark K8SSpark YARNSpark MESOS代码提交在一定程度反映了Spark社区技术趋势和热度技术趋势和热度-Kubernetes的趋势和热度在2020年超越Yarn。-Yarn、MESOS的贡献逐渐趋于平缓。Google TrendGoogle Trend在一定程度反映了用户关注度、落地程度用户关注度、落地程度。-Spark K8S的关注度逐渐上升，而Yarn则在下降。|Spark on K8S的时间线2020.08移除移除experimentexperime

4、nt2021.01Spark K8S GASpark K8S GA开始孵化开始孵化2016.112017.12正式合入正式合入Spark 2.3Spark 2.32017.08投票通过投票通过2016.11 2016.11 提交了第一PR SPARK-18278。社区决定在apache-spark-on-k8s repo孵化。2017.08 2017.08 Spark on Kubernetes SPIP VOTE 通过。2017.122017.12 Spark on Kubernetes在Spark的代码正式合入Spark 2.3。SPARK-18278 SPARK-226462020.08

5、 2020.08 官方文档中正式移除“实验”标记，并认可3.1 K8S GA：SPARK-325542021.01 2021.01 Spark 3.1.1正式发布，Kubernetes正式GA SPARK-33005|Spark on K8S的主要方案Native Spark App in K8SSpark on Kubernetes Operator-Apache Spark官方实现-提交方式：./spark_submit-通过-conf配置Kubernetes的参数-Google 开源-提交方式：kubectl apply*.yaml-通过yaml配置Kubernetes的参数|Dive

6、in to Spark on K8S实现KubernetesKubernetesDriverDriver BuilderBuilderDriverspark-submit KubernetesExecutor BuilderExecutorSeq(new BasicDriverFeatureStep(conf),new DriverKubernetesCredentialsFeatureStep(conf),new DriverServiceFeatureStep(conf),new MountSecretsFeatureStep(conf),new EnvSecretsFeatureStep

7、(conf),new MountVolumesFeatureStep(conf),new DriverCommandFeatureStep(conf),new HadoopConfDriverFeatureStep(conf),new KerberosConfDriverFeatureStep(conf),new PodTemplateConfigMapStep(conf),new LocalDirsFeatureStep(conf)Seq(new BasicExecutorFeatureStep(conf,secMgr,resourceProfile),new ExecutorKuberne

8、tesCredentialsFeatureStep(conf),new MountSecretsFeatureStep(conf),new EnvSecretsFeatureStep(conf),new MountVolumesFeatureStep(conf),new LocalDirsFeatureStep(conf)Feature StepPipeline构建构建Driver Pod构建构建Executor Pod/Statefulset过去的Apache Spark 3.0带来了什么？Kubernetes EnhancementsSupport user-specified drive

9、r and executor pod templates(SPARK-24434)Allow dynamic allocation without an external shuffle service(SPARK-27963)More responsive dynamic allocation with K8S(SPARK-28487)Kerberos Support for Spark on K8S(SPARK-23257)Kerberos Support in Kubernetes resource manager(Client Mode)(SPARK-25815)Support cli

10、ent dependencies with a Hadoop Compatible File System(SPARK-23153)Add configurable auth secret source in k8s backend(SPARK-26239)Support subpath mounting with Kubernetes(SPARK-25960)Make Python 3 the default in PySpark Bindings for K8S(SPARK-24516)Core SPIP:Add better handling for node shutdown(k8s

11、decomisson):https:/issues.apache.org/jira/browse/SPARK-20624关键词：关键词：稳定性稳定性提升提升过去的Apache Spark 3.1带来了什么？Kubernetes GA(SPARK-33005)Adds support for Kubernetes NFS volume mounts(SPARK-31394)Support dynamic PVC creation/deletion(SPARK-32971,SPARK-32997)Respect environment variables and configurations fo

12、r Python executables(SPARK-33748)Support Python dependency(SPARK-27936)Make pod allocation executor timeouts configurable and allow scheduling with pending pods(SPARK-33231,SPARK-33262)Respect executor idle timeout conf in ExecutorPodsAllocator(SPARK-33099)Support JDBC Kerberos with keytab(SPARK-123

13、12)关键词：关键词：GAGA，生产可用，生产可用最新的Apache Spark 3.2.0带来了什么？Kubernetes EnhancementsUpgrade Kubernetes client to 5.4.1(SPARK-35660)Support spark.kubernetes.executor.disableConfigMap(SPARK-34316)Support remote template files(SPARK-34783)Introducing a limit for pending PODs(SPARK-36052)Support shuffle data rec

14、overy on the reused PVCs(SPARK-35593)Support early driver service clean-up during app termination(SPARK-35131)Add config for driver readiness timeout before executors start(SPARK-32975)Support driver-owned on-demand PVC(SPARK-35182)Maximum decommissioning time&allow decommissioning for excludes(SPAR

15、K-34104)Support submit to k8s only with token(SPARK-33720)Add a developer API for custom feature steps(SPARK-33261)关键词：易用性、可扩展性关键词：易用性、可扩展性即将发布的Apache Spark 3.3.0带来了什么？Kubernetes EnhancementsSPIP:Support Customized Kubernetes Schedulers https:/issues.apache.org/jira/browse/SPARK-36057Introduce pendi

16、ng pod limit for Spark on K8s：https:/issues.apache.org/jira/browse/SPARK-36052Support replicasets/job API：https:/issues.apache.org/jira/browse/SPARK-36058Support for specifiying executor/driver node selector：https:/issues.apache.org/jira/browse/SPARK-36075Add Spark appname as a label to the executor

17、 pods：https:/issues.apache.org/jira/browse/SPARK-36566Add the ability to create resources before driver pod：https:/issues.apache.org/jira/browse/SPARK-37331Add spark-version label to driver and executor pods：https:/issues.apache.org/jira/browse/SPARK-37339Executor Rolling in Kubernetes environment：h

18、ttps:/issues.apache.org/jira/browse/SPARK-37810关键词：调度能力提升、可靠性提升关键词：调度能力提升、可靠性提升Spark 3.3 Spark 3.3 自定义调度支持自定义调度支持02|HPCHPC、大数据、大数据、AI AI 批量计算的发展历程和趋势批量计算的发展历程和趋势2015TensorflowCaffe2Pytorch2017kubeflowHadoop 1.0HDFS20062008Hive2012YARNImpala2014SparkStormFlink2010HbaseNOSQL2013Tez2017Spark on k8sK8s,

19、framework,container20182010Slurm199*LSFPBSSGE2005SymphonySGE/UGE2005 2008GlobusHPC for AIisc/vhpcKubeflowkubecon2013长期课题：长期课题：调度效率，资源利用率资资 源源 池池 ：多个独立资源池 统一的大资源池业务类型：业务类型：单一作业平台 在离线作业混部技技 术术 栈栈 ：自建底层技术栈 拥抱云原生底座|SPIP：Support Customized Kubernetes Schedulers2022.01SPIPSPIP投票投票通过通过2022.033.3 Feature Fr

20、eeze3.3 Feature FreezeVolcanoVolcano进入进入主线主线社区社区讨论讨论2021.062021.11发起发起SPIPSPIP讨论讨论2021.07JIRAJIRA创建创建2021.062021.06：邮件列表讨论希望能够支持更完善的调度能力:Spark on Kubernetes scheduler variety2021.072021.07：Volcano和Spark开发者共同发起：SPARK-36057:Support volcano/alternative schedulers2021.112021.11：正式发起SPIP讨论：DISCUSSION SPI

21、P:Support Volcano/Alternative Schedulers 2022.012022.01：正式发起投票：VOTESPIP Support Customized Kubernetes Schedulers Proposal，1周后投票通过，来自Netlfix、Apple、Cloudera、AWS、Nvidia、Linkedin、网易等厂商的支持。2022.032022.03：SPARK-36057 最终在Spark 3.3版本合入，正式可用（Experimental）。Volcano代码全部合入，同时支持X86/ARM64多架构。|Spark自定义调度支持：我们做了什么？C

22、ommon part（框架能力部分，3.3版本提供）:New configuration:scheduler name New workflow：create pre-populated resource before driver podNew developer API：user feature step with configurationNew annotation placeholder:Support APP_ID and EXECUTOR_ID placeholder in annotationsNew doc:Add doc for Customized Kubernetes

23、SchedulersVolcano part（作为参考实现，3.3版本Beta可用）:New profile:-PvolcanoNew volcano extension in kubernetes-clientNew Feature Step:VolcanoFeatureStepNew integration test:Test basic volcano workflowNew Configurations:spark.kubernetes.scheduler.volcano.podGroupTemplateFile|支持指定自定义调度器spark.kubernetes.executor.

24、scheduler.namespark.kubernetes.driver.scheduler.namespark.kubernetes.scheduler.name|New workflowNew workflow：创建预置资源的：创建预置资源的能力能力driverPodOther resource with driver owner referencedriverPodcreateOther resource with driver owner referencepre-populated resource createrefresh pre-resource owner referenc

25、etrait KubernetesFeatureConfigStep def configurePod def getAdditionalPodSystemProperties def getAdditionalPreKubernetesResources def getAdditionalPreKubernetesResources新增前置资源创建能力创建前置资源的新接口SPARK-37331:Add the ability to create resources before driverPod creating|New developer APINew developer API：动态加

26、载Feature Step-Support KubernetesDriverConf/KubernetesExecutorConf-Support KubernetesConf一个用于Driver及Executor的动态Feature StepSPARK-37145:Support the featurestep with configuration|New annotation placeholderNew annotation placeholdersupport APP_ID and EXECUTOR_ID placeholder in K8s annotation.set(spark.

27、kubernetes.driver.annotation.xxx.yyy.xxx/app-id,APP_ID)|New doc:新增文档JIRA:SPARK-38561Spark 3.3 VolcanoSpark 3.3 Volcano集成集成设计与设计与实现实现03|VolcanoVolcano的整体的整体架构架构特性特性 1 1:支持面向高性能负载的调度策略,e.g.fair-share,gang-scheduling特性特性 2 2:支持多种作业生命周期管理,e.g.multiple pod template,error handling特性特性 3 3:支持多种异构硬件,e.g.GPU

28、,FPGA特性特性 4 4:面向高性能负载的性能优化，e.g.scalability,throughput,network,container runtime|Volcano的批量调度能力Pod GroupQueue-以Pod Group为单位调度-最小资源最小资源（Pods/CPU/MEM）-调度优先级-引入队列概念-多租户-多队列优先级-更多的K8S原生资源原生资源支持:StatefulSet/Job.|Spark+Volcano最终的用户接口：FEATURES=”org.apache.spark.deploy.k8s.features.VolcanoFeatureStep”bin/spa

29、rk-submit -master k8s:/https:/127.0.0.1:60250 -deploy-mode cluster -conf spark.executor.instances=1 -conf spark.kubernetes.scheduler=volcano -conf spark.kubernetes.driver.pod.featureSteps=$FEATURES -conf spark.kubernetes.executor.pod.featureSteps=$FEATURES -conf spark.kubernetes.scheduler.volcano.po

30、dGroupTemplateFile=/path/to/podgroup-template.yaml -conf spark.kubernetes.namespace=spark -conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa -conf spark.kubernetes.container.image=spark:latest -class org.apache.spark.examples.SparkPi -name spark-pi local:/opt/spark/examples/jars/

31、spark-examples_2.12-3.3.0-SNAPSHOT.jar1.Specify custom scheduler2.Specify custom feature step3.Specify scheduler hints(podgroup template)|New profile:New profile:Volcano，新的Spark模块Built-in Module vs Profile-Built-in module:https:/ all classes-Without-Pvolcano,filter out all volcano classes.SPARK-3606

32、1:Add volcano module and feature step|Volcano Java/Scala SDK in kubernetes-clientVolcano extension available since k8s-client v5.11https:/ same user experience with K8S Java SDK.val podGroup=new PodGroupBuilder().editOrNewMetadata().withName(podGroupName).withNamespace(namespace).endMetadata().editO

33、rNewSpec().withQueue(queue).withMinResources(minRes).withPriorityClassName(priorityClassName).endSpec().build()client.podgroups().createOrReplace(podGroup)val volcanoClient=kubernetesClient.adapt(classOfVolcanoClient)/list by volcano clientvolcanoClient.podGroups().inNamespace(default).list()/create

34、 by kubernetes client directlykubernetesClient.resourceList(podgroup).createOrReplace()/easy to create podgroups in exsiting workflow kubernetesClient .resourceList(preKubernetesResources:_*).createOrReplace()SPARK-37258与Kubernetes资源创建一样的体验：|更灵活的配置项:Volcano PodGroupTemplateJIRA:SPARK-38455This is a

35、simpler,more extensible and robust way to support Volcano future/path/to/pg.yamlapiVersion:scheduling.volcano.sh/v1beta1kind:PodGroupspec:minMember:1 minResources:cpu:4 memory:16Gi priorityClassName:executor-priority queue:executor-queuespark.kubernetes.scheduler.volcano.podGroupTemplateFile=/path/t

36、o/pg.yaml通过使用PodGroup Templates灵活配置调度所需的各种参数|VolcanoVolcano专属的专属的Feature StepFeature Step#Pod(driver and executor)apiVersion:v1kind:Podannotation:scheduling.k8s.io/group-name:spark-appid-podgroup#PodGroup()apiVersion:scheduling.volcano.sh/v1beta1kind:PodGroupmetadata:name:spark-appid-podgroupspec:qu

37、eue:default minAvailable:1 minResources:cpu:3 memory:2048Mi priorityClassName:high-prioritySPARK-36061:Add volcano module and feature stepDemoDemo演示演示04|Volcano实测结果Spark-sql-perf(TP-DCS,master)104 queries concurrently(8cpu,64G,1600SSD)*4nodesKubernetes 1.13Driver:1cpu,4G;Executor:(1cpu,4G)*5如果没有固定的d

38、river节点，最多同时运行最多同时运行 26 条查询语句条查询语句由于Volcano提供了作业级的资源预留，总体性能提高了30%Volcano(min-res):3.3 CPU,12G MemoryKubernetes:1 node for drivers|Spark+VolcanoSpark+Volcano集成集成测试测试#Deploy Volcano(x86)kubectl apply-f https:/ integration testbuild/sbt-Pvolcano-Pkubernetes-Pkubernetes-integration-tests-Dtest.exclude.t

39、ags=minikube,r kubernetes-integration-tests/testAll existing tests are passed by using volcanoSPARK-36061:Add volcano module and feature step|DemoDemo：队列能力队列能力展示展示Case 1Case 1：Step 1:Step 1:创建2个队列，1个开启开启，1个关闭关闭。Step 2:Step 2:在2个队列中，并发提交4个Job。Step 3Step 3:检查Job状态，2个成功成功，2个PendingPending1324Pendingsuc

40、ceeded24succeededCase 113succeededenabledisableenableenableJIRA:SPARK-38188 Code:linkCase 2Case 2：Step 1:Step 1:创建2个队列，2个均开启开启。Step 2:Step 2:在2个队列中，并发提交4个Job。Step 3Step 3:检查Job状态，4个Job全部成功成功。Case 2 build/sbt-Pvolcano-Pkubernetes-Pkubernetes-integration-tests-Dspark.kubernetes.test.deployMode=minikub

41、e-Dspark.kubernetes.test.namespace=default kubernetes-integration-tests/testOnly-z SPARK-38188|DemoDemo：资源预留能力：资源预留能力展示展示Step 1:Step 1:创建一个资源受限资源受限的队列。（例如queue.spec.cpu=2.0）Step 2:Step 2:并发提交Job到队列中，其中Job最小预留资源为队列资源总数（例如podgroup.spec.minResources.cpu=2.0）。Step 3Step 3:资源预留功能会为Executor预留资源，防止集群可用资源被所

42、有driver占用而出现死锁。Step 4:Step 4:所有Job依次成功1234RunningPendingJIRA:SPARK-38187Code:link1234RunningPending1234CompetedCompeted build/sbt-Pvolcano-Pkubernetes-Pkubernetes-integration-tests-Dspark.kubernetes.test.deployMode=minikube-Dspark.kubernetes.test.namespace=default kubernetes-integration-tests/testOn

43、ly-z SPARK-38187|DemoDemo：优先级调度能力：优先级调度能力展示展示Step 1:Step 1:创建一个仅能容纳一个Job的队列，先关闭Step 3Step 3:并发提交3个Job，并设置高、中、低三个优先级Step 4:Step 4:开启队列Step 5:Step 5:三个Job按照高、中、低优先级顺序依次成功HLMRunningPendingHMLSucceeded one by one:High-Medium-lowJIRA:SPARK-38423Code:link build/sbt-Pvolcano-Pkubernetes-Pkubernetes-integration-tests-Dspark.kubernetes.test.deployMode=minikube-Dspark.kubernetes.test.namespace=default kubernetes-integration-tests/testOnly-z SPARK-38423|New CI:New CI:Spark+Volcano IT-Volcano community:https:/ Spark+Volcano IT will be checked before each volcano PR merged.非常感谢您的观看非常感谢您的观看|