1、网易 马进ArcticArctic：基于：基于Flink+icebergFlink+iceberg 的湖仓一体的湖仓一体Arctic:Using flink and iceberg to build netease lakehouse背景与目标背景与目标#1Background and goalsArctic Arctic 特性特性#2Arctic featuresArctic Arctic 架构架构#3Arctic architecture总结总结#4Summary#1#1背景与目标背景与目标Background and goals离线数据中台体系数据传输batch transferbatc

2、h tableolap systemBI/AI businessspark/prestohivespark/hivespark/hiveimpala/sparksql数据开发数据资产数据血缘数据质量数据分析有数数据模型任务运维Timing scheduling drive数据服务实时数仓体系cdc/event fetchstream tableolap systemBI/AI businessflinkcdc/canal/ndckafkaflinkimpala+kududruid/doris/clickhouseredis/mysql/oracle实时开发实时血缘实时任务监控运维flinkba

3、tch transferbatch tableolap systemBI/AI businessTiming scheduling driveEvent drive存在的问题主题域/数据分层modelDQCpropertiessensorslogsdatabases数据源demanddemanddemand数据治理VS点对点开发datalakedata governingp2p developing目标：实时数据中台batch transferfused tableolap systemBI/AI businessspark/hive/flinkspark/flinkimpala/sparks

4、ql数据开发数据资产数据血缘数据质量数据分析有数数据模型任务运维数据服务cdc/event fetchTiming scheduling driveEvent drive流批一体目标拆解存储流批一体开发流批一体工具流批一体统一 schema统一存储引擎统一存储介质消除二义性使用一套代码覆盖实时和离线场景统一 UDF统一开发规范数据模型数据资产数据质量数据血缘数据传输one data fits allone tool fits allone code fits allArctic 需求支持基于主键的流式更新支持流式读/增量读（Stream and CDC）支持各类引擎的并发读写，提供ACID 保

5、障提供分钟级数据延迟的 olap 能力提供湖仓一体服务，而不是软件库support streaming update based on primary keysupport streaming/incremental read(stream and CDC)support concurrent reading or writing with ACIDprovide minute latency olap abilityprovide lakehouse service,not libs#2 2Arctic Arctic 特性特性Arctic featureschange file（insert

6、/update/delete）base file(insert)tmp filenew base tablechange tablebase tabletmp filesSTEP 1:Hive MR 增量传输方案map and shufflereduceIO 效率低（IO/有效更新）写放大与实时性 trade off没有 ACID 保障适合小时级别的增量同步low io efficiencywrite amplification vs data latencyno ACID guaranteesuited for hour latencySTEP 2:基于 Bucket 数据整理方案chang

7、e tablebucket hashbucket 0bucket 1bucket 2bucket 3base filesnew base tablebucket 0bucket 1bucket 2bucket 3PKingestion hashed by 4 去 shuffle，提升 IO 效率remove shuffling,improve IO efficiencyminorcompactionmajorcompaction 谁设 bucket？怎么设 bucket?Who and how to config bucketsArctic tree 数据组织原理Arctic Tree 数据整

8、理方案change tablenode hashbase filesnew base tablenode 3node 4node 5node 6node 7node 8node 9node 10node 11node 12node 13node 14node 7node 8node 9node 10node 11node 12node 13node 14PKminorcompactionmajorcompaction 动态分片，灵活扩展Auto bucketing and extendingingestion hashed by 4 解决了谁设，怎么设bucket的问题resolve buck

9、et config probleminsert filebase filechange tablebase tablenode 3node 4node 5node 67Arctic Tree 读时合并891312111410delete file动态分片保障 IO效率读时合并提供实时性保障（分钟级）后台合并治理小文件，业务无感delete/insert 拆分提升 MOR 性能Arctic 提供多引擎适配auto bucketing improve IO efficiencymerge on read provide low latency(minutes)background compacti

10、on reduce file fragments without business concerningsplit delete and insert files to improve merge on read performancearctic provide presto/impala/presto/flink adaption#3 3Arctic Arctic 架构架构Arctic architectureMetaCacheGoverningPlannerTaskDispatchercompactingsplittingsortingArctic MetaServiceminor co

11、mpactionmajor compactionindex compactionArctic AMS 架构Arctic web servicechange filebase filetmp fileArctic 流批一体解决方案key indexChange dataBase datakey indexkey indexChange dataBase datakey indexChange dataBase datasensorslogsdatabasesOLAP merge on readkey indexChange dataBase datachange database datadat

12、a sourceArctic ODSArctic 维表Arctic DWDArctic DWSKV 存储flinkspark/flinkflinkflinksparksparkMetaCacheGoverningPlannerTaskDispatchercompactingsplittingsortingArctic Meta Serviceminor compactionmajor compactionindex compactionArctic AMS 架构Arctic web servicechange filebase filetmp filesync commitstreamstre

13、amstreamstreamstreamstreamstreambatchbatchstreamPropertiesMapModelFused datalakeFused tool stackHive metastoreYARN/K8SbatchbatchArctic meta serviceArctic engine grouppresto/impalaspark/hiveflinkFused lakehouseDevelopQualityTools layer：data dev and op adaption data governing adaptionHouse layer：arcti

14、c meta service,hms adaption stream and batch file governing support presto/impala/spark/hive/flink 数据湖层：基础元数据管理 MVCC/ACID 保障 快照管理，数据回滚工具层：数据研发和运维适配 数据治理适配数仓层：元数据管理服务，对接 hms 流批文件治理服务 各类计算引擎对接Lake layer：basic meta store provide MVCC/ACID function snapshot management,rollbackTransferArctic 架构#4 4总结总结Su

15、mmarystreamstreamstreamstreamstreamstreamstreambatchbatchstreamPropertiesMapModelFused datalakeFused tool stackHive metastoreYARN/K8SbatchbatchArctic meta serviceArctic engine grouppresto/impalaspark/hiveflinkFused lakehouse serviceDevelopQualityTransferArctic 架构 stream ingestion stream compaction M

16、VCC supporting ACID supporting snapshot management merge on read adaption batch write adaptionArctic lakehouse concepts湖仓一体=存储流批一体=基于数据湖实现所有数仓功能支持流式更新，基于主键更新实时/准实时更新数据支持实时，增量，批量三种数据读方式，且支持数据分钟级延迟ACID 保障，支持三种方式的并发写入，保障数据一致性快照和版本管理，支持版本回溯和回退Arctic 核心优势Auto bucketing and extending10 x faster than bloom

17、 filter数据摄取数据摄取stream compactionminor and major compactionbusiness no concern数据整理数据整理separates delete and insertimproves memory usagesupports delta push-down读时合并读时合并provides standard meta servicebackground table governingAMSAMS服务服务ingestioncompactionmerge on readmeta serviceFuture work多引擎一致性AP 性能multiple engine supporting business consistency concernmerge on read performance