《8-任庆盛-高谟-Flink Connector 的架构解析和最佳开发实践-已美化TY-定稿_compressed.pdf》由会员分享,可在线阅读,更多相关《8-任庆盛-高谟-Flink Connector 的架构解析和最佳开发实践-已美化TY-定稿_compressed.pdf(37页珍藏版)》请在三个皮匠报告上搜索。
1、Flink Connector Flink Connector 的架构解析和最佳开发实践的架构解析和最佳开发实践 Architecture and Development Practice of Flink Connectors 任庆盛 Qingsheng Ren 阿里巴巴 Alibaba Group 高赟 Yun Gao 阿里巴巴 Alibaba Group ConnectorConnector Testing FrameworkFramework # #2 2 SourceAPIAPI # #1 1 Sink Sink API # #3 3 SourceSourceAPI#1 开发者需要自
2、己处理复杂的线程模 型,开发和测试门槛高 Source 的任务发现、分工和读取 数据的逻辑杂糅在一起,导致如 Kafka、Kinesis 等 Source 的实现 非常复杂 与 Flink Runtime 的过度耦合,例 如 Checkpoint 锁、对算子无锁 Actor/Mailbox 线程模型的影响等 #2#2#1#1#3#3 动机 Motivation Developers have to deal with complex threading model, which adds a high bar to developing and testing Logic for work d
3、iscovery and reading the data is intermingled, leading to complex implementations like the Kafka and Kinesis source Over-coupling with Flink runtime, such as checkpoint lock, effect on Actor/Mailbox threading model FLIP-27 SourceSourceAPI Source Split Enumerator Source Reader Task Task Task Source Reader Source Reader Source Reader Split Enumerator JobMaster 分配 Splits、发送全局水位 Assign splits, send gl