《Doris Summit-平安人寿基于 Apache Doris统一 OLAP 技术栈实践-（脱敏版）.pdf》由会员分享，可在线阅读，更多相关《Doris Summit-平安人寿基于 Apache Doris统一 OLAP 技术栈实践-（脱敏版）.pdf（21页珍藏版）》请在三个皮匠报告上搜索。

1、平安人寿基于 Apache Doris 统一 OLAP 技术栈实践孙顺平安人寿 大数据架构师Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

2、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目录1 大数据平台建设沿革与总览2 早期架构与应用痛点3 基于 Apache Doris 统一 OLAP 技术栈的演进之路4 总结与未来规划Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

3、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

4、sia 20231大数据平台建设沿革与总览Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

5、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023大数据平台建设历程20052022至今2009大数据产品体系数据仓库与数据集市1.集中业务系统数据2.按需开发数据报表统一 OLAP 技术栈、统一指标与标签管建、统一数据服务大数据 Hadoop 平台1.存储力和算力大2.提升报表刷新时效2016设计数据中台1.数据治理2.数据底座3.数据产品Doris Summit Asia 2023Doris Summit A

6、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

7、Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023大数据产品体系总览Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

8、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20232早期架构与应用痛点Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

9、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023早期架构总览Doris Summit Asia 2023D

10、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

11、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023早期应用痛点报表分析HivePostgreSQL即席查询HivePrestoPostgreSQL总部业务运营人员管理层经营数据探查痛点升级需求业务洞察问题定位趋势预测经营全貌概览多轮清洗计算，数据链路长PG 扩容成本高导数冗余、资源浪费应用层需配合改造大规模数据处理能力报表产出时效性报表查询响应在百毫秒内资源成本控制客户经营产品、理赔、核保等数据可视化分析指标数据集订阅痛点查询性能不及预期Hive 权限管理不清晰缺少资源隔离限制升级需求支持高并发查询

12、支持低延迟响应指标复用率Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

13、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023早期应用痛点人群圈选ClickhouseKylin升级需求CK 难以支撑 200 人并发查询组件语法差别，适配成本高依赖人工判断、灵活度不足运维成本高多维分析Druid一线业务运营人员高并发查询访问报表查询响应时间指标开发扩展性精确去重与关联查询痛点业务运营决策业绩报表实时数据监控和分析节点重启时间超 24 小时查询灵活度低非精确去重运维成本高升级需求总部与分公司营销人员人群圈选画

14、像分析人群检测与追踪痛点支持高并发查询多表关联查询能力数值间圈选Array 字段Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

15、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023早期数据开发系统痛点标签数据开发组件 HBase指标数据开发组件 PG Presto Druid 1.平台组件不同，数据未打通2.一个需求配置一个接口，无法动态配置与灵活调用痛点 依赖预计算 不同组件支持指标存储与查询 不支持复合指标开发痛点 不支持二级索引 日期、数字需额外转化 不支持标签跨对象Doris Summit As

16、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

17、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023架构升级目标强化查询性能丰富查询与写入的优化功能打通平台数据读取，将指标与标签数据统一存储，实现复合指标加工数据治理体系化、数据一致性覆盖不同应用系统的日常分析需求，面向更多样化的业务场景综合性强、灵活度高有效简化数据链路、架构技术栈，降低对技术人员开发的依赖开发、存储、运维成本控制高效写入与极速分析持续构建一站式数据门户Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

18、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

19、 Summit Asia 2023Doris Summit Asia 20233基于 Apache Doris 统一 OLAP 技术栈的演进之路Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

20、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023为什么选择 Apache Doris 高效的写入性能极速的查询分析简单易用的架构 支持主键模型写时合并 支持 Schema 在线变更 支持 Upsert 所有导入方式，条件更新、条件删除、部分列更新以及分区覆盖 支持多种存储模型 支持物化视图与 Rollup 预聚

21、合 支持高并发与低延迟查询 支持多表关联查询 提供灵活的数值区间圈选 支持 Bitmap 精确去重，HyerLogLog 近似去重 采用 MySQL 协议，支持标准 SQL MPP 架构，仅 FE 与 BE 两类进程 节点间负载均衡 列式存储，极高压缩比Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

22、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023基于 Apache Doris 构建全新大数据产品体系统一 OLAP 技术栈 Doris 集群之上构建统一的指标标签设计平台 形成“上下经营一张表”完善经

23、营指标追踪体系统一的指标标签设计平台 统一的数据查询与计算引擎 可插拔引擎 减少重复开发、冗余存储 API 接口服务化 打通各类应用系统的数据调取 统一的数据服务平台Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

24、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202314倍10倍统一 OLAP 技术栈，加速业务需求交付原架构链路长、导数重复、定制开发14 个 PG 数据库成本压力业务交付耗时 2 周新架构 架构统一、运维成本降低丰富的数据模型简化链路业务交付仅需 1-2 天原架构缺少权限、表交叉使用查询分

25、钟级耗时新架构业务人员只可以访问 ADS 层数据提升数据指标复用率查询秒级毫秒级响应查询响应提速交付效率提升Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

26、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023统一指标和标签管理，直通应用场景指标和标签服务 API指标与标签定义指标与标签统一计算数仓层引擎层应用层Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

27、it Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023统一

28、数据服务平台，打破数据孤岛Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

29、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20234总结与未来规划Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

30、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023湖仓一体化基于 Doris 的统一分析架构Apache Doris湖仓（Hive，Hudi）之前的分析架构Hive湖仓HudiDruidOLAPPrestoClickhous

31、ePostgreSQLHBaseKylin更开放、灵活、可扩展的企业级管理与分析大数据产品体系Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

32、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023多租户资源隔离Resource Group 机器分组 副本放置到分组 User 绑定分组WorkLoad Group 对特定查询模式、分区/分片数大的查询拦截 单查询内存限制 多 Cluster 机制和 Resource Group 整合完善应用系统之间负载均衡性能，使集群资源更合理分配Doris Summit

33、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

34、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023获取更多社区动态与最佳实践Doris Summit 峰会官网：doris- Doris Summit 峰会回放：https:/ Doris 官网：doris.apache.orgApache Doris GitHub： Doris 官方平台：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

35、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023