1、Apache Doris 在网易互娱的管理与应用实践胡彪网易游戏 高级大数据开发工程师Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

2、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023个人介绍胡彪|网易互娱 高级大数据开发工程师 在网易互娱负责 Trino/Doris/统一查询引擎等组件的开发维护和业务支持工作 在 OLAP 引擎开发和平台建设上有一定的研究经验 Trino Contributor Apache Doris ContributorDoris Summit Asia 2023Doris

3、Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

4、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目录2.生态建设3.场景加速4.未来展望1.背景介绍Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

5、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20231背景介绍Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

6、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023背景介绍技术中心-

7、数据与平台服务部负责构建和支持迭代网易互娱的数据平台和数据基础架构。经历了从第一代离线架构到第二代 Kappa 架构演进之后，我们最终引入了Apache Doris搭建统一的湖上实时数据仓库。本次分享将详细介绍我们在三代架构演进中的Doris管理平台建设和业务场景落地的实践经验。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

8、Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023引入 Apache Doris 前的组件架构Doris Summit Asia 2023Doris Summit Asia 20

9、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

10、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023架构复杂，运维困难用户研发成本高数据时效性与查询效率较低架构简单，低运维成本统一易用，降低用户学习成本实时高效，支持实时数据导入，湖仓加速实现高效查询架构痛点目标需求痛点与需求Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

11、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023选型因素关键指标Apache Doris查询性能内部SSB数据集及业务场景测试均满足性能指标要求，1.1 版本以后提供向量化

12、执行、1.2 提供新版查询优化器，性能提升更多。导入方式支持离线和实时的导入方式，1.1 版本支持事务导入，能够做到实时数据写入不丢不重。开源方式采用 Apache License 协议，安全性高、灵活性强。社区活跃度官网文档详细，同时提供了多堂源码解读课程。社区活跃度较高，Issue 问题能够得到及时反馈。使用方式兼容 MySQL 协议，接入和迁移成本低运维部署不依赖第三方组件，FE/BE 扩缩容简单，同时支持数据自动平衡。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

13、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

14、t Asia 2023基于Doris构建高效易用的湖上实时数仓Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

15、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023集群规模当前总集群数国内/海外均有布局10+总节点数接近百台大部分为FE/BE混部100+最大的一个集群当前存储数据总量900TB+平均每日查询总数量通过实时作业和离线导入的日增数据量10TB+对接内部项目数100+2000000+Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

16、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

17、 Summit Asia 2023Doris Summit Asia 20232生态建设Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

18、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023大规模集群运维方向1.基础建设制定好运维规范，如端口、目录、服务器类型/配置制定好业务接入准则、开发规范，完善基准测试报告建设好元信息，集群、实例、库等维度，集群管理员、业务线等3.安全性保障制定好备份规范、备份方式，自动化备份与恢复各类故障演练集群巡检、规则治理2.监控报警梳理集群分级指标、设定报警规则和升级机制用户

19、侧性能指标监控统一入口，查看所有集群重点监控情况4.平台化利用自动化工具和脚本来快速、可靠地部署和配置集群开发易于使用和管理的用户界面，多租户高可靠的权限模型提供数据报表和分析功能，帮助用户了解业务数据趋势和业务指标Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

20、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Manager 架构功能特性 可视化操作：提供直观、简便的方式操作 Doris 集群，减小误操作的概率，快速上线业务。多集群管理：多个集群统一管理，提供一个集中化的控制面板，简化管

21、理流程。丰富的权限管理：将资源抽象化为部门、项目级别，角色的权限可以更细化的控制，适用于多部门、多项目的统一集中管理。完备的审计：所有操作都会有完整的审计功能，方便追踪操作历史。集成 Doris 常用功能：从平台侧帮助用户了解集群使用状态、更好地使用 Doris 服务。多层次监控报警：提供多级别的监控统计信息，对于异常情况，还能够及时发出报警通知，帮助用户快速发现和解决问题，提高系统的稳定性和可靠性。Doris 管理平台架构图Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

22、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

23、t Asia 2023Doris Manager数据流转Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

24、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023集群服务设计目标及收益目标：帮助用户全方位了解集群查询情况，提速查询当前收益：慢查询数减少 80%查询导入存储目标：帮助用户掌控实时和离线导入的数据增量情况，导入 问题报错辅助排查，减少人力运维当前收益：月均减少人力沟通次数 70%目标：帮助用户减少冗余存储，节约存储成本，减轻元数据管理压力当前收益：平均存储下降 20%Doris Summit As

25、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

26、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023查询分析 查询概览支持查看集群近3个月各指标的走势数量、CpuTime、Memory、P系列指标以表格的形式精确展示指标详情统计集群内各用户消耗资源占比所有统计指标支持导出下载设计内容价值阐述集群查询消耗的总资源和各用户消耗的资源都需要量化，以便开展用量预警、性能分析等工作。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

27、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

28、ummit Asia 2023查询分析 当前查询支持指定时间间隔刷新查看当前集群运行的查询可以 Kill 当前运行的查询、查看当前运行查询的执行计划价值阐述Doris 对于 Running Query 的可观测支持度不好，在某些场景下用户需要知道 Running Query 的情况，以及 Kill 误提交或者不合理的大查询。设计内容Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

29、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023查询分析 慢查询统计支持查看按库和按时间占比分布支持展示慢查询在时间轴上的

30、分布可按照任意时间维度、用户、库、阈值统计各用户的慢查询明细支持查看慢查询执行计划设计功能用户体验：通过治理慢查询，提升用户体验集群压力：通过慢查询治理，及时释放数据库资源，降低安全风险。SQL优化：通过慢查询治理，提高SQL质量和可维护性。成本节约：减少慢查询会减少系统的资源消耗，进而降低使用成本。价值阐述Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

31、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023查询分析 执行计划可视化操作，提供文字和图形两种执行计划，快速定位慢查询问题。解决方案用户通过慢查

32、询模块只看到了SQL，无法初步判断为何慢，需要切换终端查看执行计划。痛点Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

33、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023查询 Profile统计平台集成该功能，指导用户开启Profile信息，查看执行过程各阶段具体的资源消耗情况；在平台侧结合Profile信息给予诊断和优化指导，帮助用户优化SQL。解决方案对于一些无法通过执行计划准确判断缓慢原因的SQL查询，需要获取更详细的执行信息。官方提供了Profile功能来解决这个问题，但不是所有用户都了解或熟

34、悉这个功能的存在和用法。痛点Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

35、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023查询分析 慢查询报表每日上班前推送查询报表，报表中包含上一日慢查询分布、明细数据以及各项查询指标值。通过发送 T+1 报表，让业务洞察上一日的查询情况，及时做出相应的优化调整。解决方案对于用户而言，登录 Doris Manager 平台不是必选项，日常稳定使用集群服务时，用户可能不会上平台关注查询情况。痛点Doris Summit Asia 2023Doris Summit A

36、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

37、Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023查询 热度分析存储压力：帮助用户决策消亡数据集群优化：更少的数据量，集群更稳定价值阐述帮助用户下降 20%数据存储量当前收益Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

38、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023存储 Tablet治理Tablet 治理Tablet少Tablet多可删除表非分区表分区表用户根据业务实际情况结合热度分析直接删除重删重插的方式，减少表的bucke

39、t个数动态分区静态分区计算最佳Bucket数，修改动态分区值重删重插，根据分区推荐bucket个数重建分区Bucket异常空分区过多调整参数，减少预创建分区数重删重插，增大Bucket数历史文章：Apache Doris在网易互娱的应用实践Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

40、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023提供tablet概览界面，并根据上面提到的tablet治理类型，结合表类型及查询热度给出相应的优化措施和步骤指导。解决方案用户无法感知当前集群tablet的分布情况是否

41、合理，同时在管理员给出修改要求后，不知道该采用哪种策略去处理。痛点存储 Tablet分析Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

42、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023导入分析 导入概览离线和实时分开统计支持查看近3个月各指标走势包括作业数量、CpuTime、Memory设计功能管理员和业务用户均需要了解集群导入数据的详细情况根据导入的作业数和数据量走势辅助用户评估规模价值阐述Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

43、it Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

44、ris Summit Asia 2023Doris Summit Asia 2023导入分析 报错指南Flink Doris Connector、Seatunenl 等写入方式底层都是通过Stream Load传输数据。底层原理公司内部网络隔离相关端口未对用户开放网络隔离根据源码分析及日常收集报错整理到知识库，智能诊断用户报错。智能诊断Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

45、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20233场景应用Doris Summit Asia 2023Doris Su

46、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

47、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023质量数据中台介绍QData是网易互娱质量保障中心下属的大数据团队。从质量角度出发，针对游戏产品生命周期中的支付、奖励、性能、登录等主题业务为游戏提供实时监控、离线分析、报表等服务。简介Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

48、it Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023引擎需求DWSAGGREGATE(主要)、UNIQUEDWDDUPLICATE(主要)、UN

49、IQUE、AGGREGATEODSDIM维表层DuplicateKafkaHive数据源QData数仓日实时流数据量近百亿，并发写入数超过200，需要能够支持高并发写入；支持从Hive中快速同步大量历史数据；需要完整支持行为分析类型的函数，且P95指标不高于3s；日常有变更字段和更新数据的需求，需要引擎支持且不影响正常写入和查询；需求点：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

50、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023bitmap 查询提速场景描述游戏产品会在版本日当天放出游戏内容更新

51、或优化。数据团队需要量化玩家打开游戏时，从Patch更新到最后登录的过程的转化情况。Patch转化的数据场景指标需要针对玩家设备ID进行精确去重，数据量往往在 十亿 级别以上。问题现状直接 COUNT(DISTINCT)往往会占用大量内存和IO，并且查询时间 20s，特别是当表中有大量不同的值时，查询性能受到的影响更大，无法满足性能要求。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

52、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023bitmap查询提速将玩家设备id构建全局字典表导入到 bitmap

53、 列，或者针对明细表添加物化视图。解决方案Base表+轮转表构建全局字典Aggregate 模型：bitmap第一步：构建全局字典表由于玩家的设备id是字符串，所以需要先转化为整形，使用Hive全局字典表第二步：Agg模型新增表字段：udid VARCHAR(256)=udid_ranks bitmap BITMAP_UNION NULL第三步：改写查询COUNT(DISTINCT udid)=COUNT(DISTINCT udid_ranks)Duplicate 模型：使用物化视图第一步：构建全局字典表 由于玩家的设备id是字符串，所以需要先转化为整形，两种方式：1.Hive全局字典表 2.

54、使用Doris函数bitmap_hash64第二步：Duplicate模型创建物化视图：bitmap_union(TO_BITMAP(udid_ranks)第三步：改写查询 COUNT(DISTINCT udid)=COUNT(DISTINCT udid_ranks)Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

55、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023优化收益14亿数据峰值内存查询时间 优化前54.0GB20s优化后4.2GB2sDoris Summit Asia 2023Doris Su

56、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

57、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023物化视图提速查询问题描述游戏性能是玩家游戏时最直观的体验,合适的性能可以确保游戏流畅度、响应速度和画面质量。性能问题可能导致卡顿、延迟或崩溃，严重影响玩家满意度和游戏口碑和留存。因此数据团队需要对玩家游戏时的性能数据进行进行监控和分析。衡量游戏性能相关的数据指标有很多，例如：FPS、卡顿次数、内存峰值等8种，单独一个指标相关的维度更多达10个。游戏策划希望在网页端可以针对多种指标和多个维度进行自定义聚合查询，查询响应时延需要控制在2s内。Doris Summit

58、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

59、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023针对常用的维度设计物化视图，可以满足用户绝大部分自定义聚合查询的需求。物化视图提速及收益解决方法列名类型列类型时间VARCHAR(255)KEY列数据统计量LARGEINTSUM聚合列FPS 统计值BIGINTSUM聚合列FPS 最大值DECIMAL(9,0)MAX聚合列低FPS统计值BIGINTSUM聚合列内存占用最大值LARGEINTMAX聚合列Doris Summit Asia 2023Doris Summit Asia 2

60、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

61、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023基于自研大模型问答和拖拽式生成查询Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

62、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023基于 Seatunnel 的数据集成Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

63、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023使用中遇到的问题关联 Issue-1

64、3070，TransactionState 缺少了预提交状态1.1.3 版本修改 VARCHAR 长度导致 tablet 出现 missing_rowsets，表不可查 关联 Issue-13064，分隔符指定为多个时每批次会丢失最后一行数据1.1.3 版本 Flink 写入会丢失最后一行数据原因：用户使用 Unique 模型进行查询的时候，进行了两次聚合操作，因此实际上没有用到分区过滤的特性。解决方法：升级集群，借助 1.2 版本的 Merge On Write 特性，使得查询能够使用索引。1.1.5 版本简单查 TB 级大表分区，做了分区过滤的情况下仍然出现 IO 打满的情况改造相关源码，

65、支持 Hive UDF 的重载1.2.4 版本 JAVA UDF 不支持 Hive UDF 的重载升级 Broker 依赖的 Hadoop 版本至 3.x1.1.5 版本不支持导入Hadoop EC 后的数据Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

66、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20234未来展望Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

67、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

68、3未来规划引入更为廉价的存储介质以降低成本，数据湖场景下更灵活地弹性部署存算分离架构目前有使用 ES 的用户对该功能比较感兴趣，官方介绍在存储及查询速度上均有大幅提升倒排索引应用利用 Doris 的外表物化视图加速数据湖上的查询数据湖分析提供 Doris on K8S 小实例部署模式，降低用户在专属集群需求上的接入门槛Doris Manager 对于 2.0 版本新特性的管理支持，如跨集群数据同步等功能Doris Manager建设Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

69、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

70、mmit Asia 2023获取更多社区动态与最佳实践Doris Summit 峰会官网：doris- Doris Summit 峰会回放：https:/ Doris 官网：doris.apache.orgApache Doris GitHub： Doris 官方平台：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023