1、Celeborn社区支持Flink最新进展熊佳树阿里云-数据湖构建与分析Outline架构及协议设计基本架构概念统一与设计01核心功能Shuffle生命周期管理网络/读写02性能与稳定性读写性能优化容错与流控设计03进展与未来展望开发进展未来规划0401-基本架构FlinkClientImplDriver ManagerReadClientHa MasterWorkerWorkerWorkerWriteClientCelebornFlinkread/writeslotsregister|split|reserve|release|commitShuffle概念设计FlinkDataSetId(

2、16)PartitionNum(16)ProducerId(16)SparkShuffleIdMapIdAttemptIdCelebornShuffleId(int)MapId(int)AttemptId(int)AppIdJobIDAppIdMapPartition&Reduce PartitionReduce PartitionPush Shuffle+Partition 数据聚合Map Partition受限Flink场景及目前Shuffle接口必须实现部分场景具备优势Map#1Map#2Map#0Partition#1Partition#2Partition#0Sub(1.N)Sub(

3、1.M)Sub(1.T)Map Partition（Mapper）Result PartitionPartition#1Partition#2Reducer#0Map#1Map#2Map#0Partition#1Partition#2Partition#0Sub（M）Reducer Partition（Mapper）Result PartitionPartition#1Partition#2Reducer#0Flink ShuffleMapper#1Partition#1Mapper#2Partition#2Reducer#1Mapper#1Partition#1Mapper#1Partiti

4、on#1Mapper#2Partition#2Reducer#3Reducer#4Mapper#1Partition#1Mapper#2Partition#2Reducer#2Reducer#1Reducer#1Reducer#2Reducer#2协议支持-Result PartitionMap#0_1Map#0_2Map#0_0Partition#(1,_)Partition#(2,_)Partition#(0,_)Map#0_1Map#0_2Map#0_0Map Partition（模式）Reduce Partition模式Celeborn PartitionMap Partition区分

5、相同Map不同attempt的结果:PartitionId(Int)=(8 AttemptId)+(24 MapId)Reduce Partition无需区分Partition:PartitionId=ReducerID02-基本功能资源分配与管理Register Shuffle(Slot allocator)ReplicaPartition Spit/Revive/DestroyShuffle Result GroupShuffle ExpireApplication Expire网络传输控制流/数据流数据读写Flink Client/Map Partition reader/writer/

6、CommitCommit 插件Client 插件Reader 插件Writer 插件MapPartitionReducePartition03-关键技术性能支持客户端压缩&小包合并（LZ4/ZSTD）支持IO Scheduling支持Broadcast优化稳定性流控（Credit-based）内存管理Channel复用容错支持客户端压缩&小包合并（LZ4/ZSTD）减少网络IO减少文件SizeMap(Producer)CompressCelebornWriterReader/IOschedulingMerge/SplitReducer(Consumer)DecompressSerializeD

7、eserializeSplit/MergeStorage/IO schedulingSP1SP2SP3SP1SP2SP3Data File RegionRegion1Region2I1I2I3I1I2I3File Offset/Data lengthDataSP1SP2SP3Region3I1I2I3Sorted ReaderIO requestsReader SchedulerReducer#0Reducer#1Reducer#2容错效率更高性能效率受资源影响Broadcast优化Map#0MapPartition输出到不同的result Partition，下游task读取时合并SP1Re

8、gion1Region2I1I2I3I1I2I3SP1Region3I1I2I3SP1Reducer#0Reducer#1Reducer#203-稳定性 Celeborn Master(HA)多副本机制LifecycleManager Credit-Based(流控)心跳&垃圾回收机制 资源复用ClientTcp链接FileChannelCeleborn(Reader)Credit-Based流控机制Reducer(ReadClient)CreditBufferPoolBufferPoolRead NotifySendBacklogConsumer123457604-开发进展2022.11启动2023.02开发完成2023.4Alpha版本2023.5Stable版本未来计划 多副本支持 Split支持 网络重传 动态的内存管理 Hybrid Flink Shuffle支持 Mixed Celeborn Shuffle Type 支持