1、Apache Doris 在物业城市的场景实践万物云 大数据研发专家王凯 Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

2、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目录2.架构演进3.实践应用4.总结展望1.业务背景Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Sum

3、mit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20231业务背景Doris Summit Asia 2023Doris Summit Asia

4、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

5、it Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023物业城市致力于城市空间智慧运营服务，整合物联网、云平台、大数据等多重力量，合力构建新型数字空间底座，在遵循市场化的原则下，通过数字化、机械化、专业化的运营手段打通孪生的数字空间与线下实体空间的联结。通过线上线下运营，智慧化地服务包括站城一体化、产城融合等多业态在内的城市复合空间，极大提升城市治理效率。目前物业城市的实践探索已经包括横琴粤澳深度合作区【放管服改革后，解决人少事多】、成都高新区【品质逐年提升 成本逐年下降】、厦门鼓浪屿【碎片化整合，高效应急处突】、武汉市江汉区【街区+社区

6、，探索老旧小区长效运营】、深圳沙头街道【城中村共建共治】等。公司介绍Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

7、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20232架构演进Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

8、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023架构演进Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

9、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

10、Asia 2023Doris Summit Asia 2023架构演进Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

11、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023旧有架构存在如下问题：旧有架构部署成本高，查询响应慢 维护组件多，运营成本高 数据没有明确分层，很多开发都是烟囱式开发，计算口径不统一架构演进Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

12、it Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023 实

13、时看板（Dashboards），符合物业城市大脑实时大屏展示的需求 面向企业内部分析师和管理者的报表，符合物业城市实时报表的需求 即席查询（Ad-hoc Query）统一数仓构建 数据湖联邦查询 极简运维，只需要维护FE BE两个进程，不需要引入额外组件架构演进Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

14、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023架构演进-现有架构Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

15、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

16、 Asia 2023Doris Summit Asia 2023基于上述架构构建了以 Doris 为核心的物业城市实时数据底座，极大的提高了数据时效性，数据可见性从原来的 T+15min，缩短到了毫秒级全域智能运营平台车辆轨迹数据从原来的数据库 3 个月可见到现在的历史可见，查询速度成倍提升全域智能运营平台工单数据报表实时性得到了成倍提升构建了基于 Doris 的城市大脑基于 Doris 构建的实时数仓，由于不需要额外维护其他组件，极大降低了维护成本架构演进-基于 Doris 的实时数仓的改进Doris Summit Asia 2023Doris Summit Asia 2023Doris S

17、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

18、3Doris Summit Asia 2023Doris Summit Asia 2023基于 Doris 的全域智能运营平台Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

19、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023 基于 Doris 的实时数仓构建的数据底座，支撑了全域智能运营平台，城市大脑实时大屏，实时报表以及统一对外数据接口、数据推送平台，目前核心数据服务基本都迁移到了 Doris 工单、车辆轨迹等业务，以前线上服务几乎不可用，现在线上服务可以达到秒级甚至毫秒级响应，极大提升了用户体验

20、由于 Doris 只需要维护 FE 和 BE，不需要依赖额外组件，运维非常高效 Doris支持 Rollup、物化视图，基于物化视图构建的实时数仓，不仅可以对原始明细数据的任意维度进行分析，也能快速的对固定维度进行分析查询基于 Doris 的实时数仓的优势Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

21、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20233实践应用Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

22、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

23、023Doris Summit Asia 2023实时大屏Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

24、ummit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023工单Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

25、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023 工单表创建使用基于时间的按月动态分区 开启了写时合并，消除掉了读时合并中的数据聚合过程 新增 BloomFilter 索引，BITMAP索引，加速查询速

26、度 基于上述优化，工单的查询速度较之前提升了 5 倍左右工单建表优化Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

27、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris 在使用一段时间后，会出现大量慢查询，导致生产不可用，通过排查以后发现如下问题：建表分区、分桶不合理，很多表只使用了 Doris 默认索引，而没有使用 BITMAP 等索引来加速查询业务方 SQL 不规范，出现大量 select*写法数仓分层基本很多依赖视图，导致谓词不能下推使用 API 写入报错Doris 优化Doris Su

28、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

29、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023基于上述问题，在使用过程中做了如下优化：增加 BITMAP 索引，加速查询设置合理的分区分桶优化 SQL 写法增加慢查询监控增加 fe be节点自动拉起将部分多表视图放到 Flink 中计算后写入 Doris，直接基于 Doris 查询Doris 监控及预警，及时发现doris的慢查询等问题Doris 优化Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

30、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

31、Summit Asia 2023Doris Summit Asia 2023Flink CDC 使用 API 写入数据报错，与社区同学沟通，判断是 Doris 1.2.1 连接器 bug，需要在字段映射里面新增隐藏的逻辑删除标识Doris 优化Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Do

32、ris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris 数据备份到华为云Doris 数据备份Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

33、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

34、Summit Asia 2023Doris Summit Asia 20234总结展望Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

35、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023升级 Doris 2.0+全新查询优化器 倒排索引 点查询并发能力提升 20 倍 自适应的并行执行模型 高效的数据更新总结展望Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

36、it Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023获取

37、更多社区动态与最佳实践Doris Summit 峰会官网：doris- Doris Summit 峰会回放：https:/ Doris 官网：doris.apache.orgApache Doris GitHub： Doris 官方平台：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023