1、实时数据更新与极速查询兼顾：Apache Doris Unique Key 写时合并实现揭秘张晨飞轮科技 资深技术专家Apache Doris CommitterDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

2、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023个人介绍拥有超过10年的大数据行业经验，曾先后在微软、百度、小米、蚂蚁任职，在OLAP系统、HSAP系统、分布式文件系统、对象存储领域都有深入的研发经历。张晨SelectDB资深研发工程师Apache Doris CommitterDoris S

3、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

4、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目录1.OLAP 系统的实时更新需求介绍2.OLAP 系统数据更新的挑战3.实时数据更新与极速分析如何兼得4.Doris 2.0 数据更新能力全面解析5.用户实践Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

5、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20231OLAP 系统的实时更新需求介绍Doris Summit Asia

6、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Sum

7、mit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023分析性能越来越快天级小时级分钟级秒级大数据量的分析延迟越来越低天级小时级大数据量的更新延迟还有很大的提升空间Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

8、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023OLAP 系统中的数据更新需求普遍存在MySQL CDC电商交易订单广告效果投放营销业务报表Doris Summit Asia 2023Dor

9、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

10、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023对数据更新的实时性要求越来越高流计算引擎TP 数据库Bin LogDoris包裹订单状态更新数百个仓库数千QPS实时调度实现当日达仓库包裹调度数据更新10万条/s 的峰值写入流量20并发同时写入实时更新，数据新鲜度 10s 以内保留半年以上数据，总数据量百亿条数据查询百毫秒级的查询延迟数千QPSAd-Hoc Query几乎所有的查询都涉及多张宽表的join业务平台某头部物流公司的场景案例Doris Summit Asia 2023Doris Summit

11、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

12、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20232OLAP 系统数据更新的挑战Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dor

13、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023OLAP 数据更新的挑战Napa:Powering Scalable Data Warehousing with Robust Query Performance at Google Doris Summit Asia 2023Doris Summit

14、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

15、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023OLAP 数据更新的挑战利用压缩、编码来减少文件大小，难以修改丰富的索引，更新成本高常见的更新方式（MoR，CoW）性能代价较大列存数据更新的挑战按列存储，可以只读取需要的列数据按类型高效的进行编码和压缩丰富的索引和裁剪OLAP系统往往采用列存大大减少一次分析需要读取的数据量Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

16、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023大数据常见

17、的数据更新方式：Merge-on-Read行存系统（KV系统）通过MVCC来实现高并发的写入支持通过merge-on-read来支撑较高的点查QPS代表系统如：LevelDB、RocksDB、HBase等列存系统（OLAP系统）很多列存系统并不支持更新支持merge-on-read的代表系统有：Hudi、Iceberg、ClickHouse 的 ReplacingMergeTree、Apache Doris的Unique KeyDoris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

18、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

19、it Asia 2023OLAP 系统使用 Merge-on-Read 方案的性能问题-例子：查询指定日期范围内所有账单开销的总额SELECT SUM(fee)FROM ledgerWHERE bill_date=2023-08-01 00:00:00 AND bill_date 16s消费金融场景半年的物流运单数据分析，宽表 200 多个字段，MoW8 并发 flink connector 任务做 upsert，10s checkpoint，吞吐 6w行/s物流运单分析场景40 并发 flink connector 任务做 upsert，10s checkpoint导入稳定，吞吐可达 7.4

20、9w行/s某客户 PoC 压力测试支持月统计（数据规模千亿左右）、年统计（数据规模万亿左右）upsert 吞吐 10w行/s 支付订单分析场景Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Sum

21、mit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023获取更多社区动态与最佳实践Doris Summit 峰会官网：doris- Doris Summit 峰会回放：https:/ Doris 官网：doris.apache.orgApache Doris GitHub： Doris 官方平台：Doris Summit

22、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023