1、奇安信基于 Apache Doris 的日志安全分析系统升级实践舒鹏奇安信 服务端技术专家Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

2、Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目录2 架构升级之旅3 Apache Doris 2.0 查询提速实践经验4 规划与展望1 公司介绍与业务背景Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

3、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20231公司介绍与业务

4、背景Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

5、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023奇安信科技集团股份有限公司中国企业级网络安全市场的领军者专注于为政府和企业用户提供新一代网络安全产品和服务。2023 年，奇安信在数据安全领域全面发力。目前核心产品天擎终端安全系统在国内已有 4000 万政企用户部署、全国部署服务器超过 100 万台、服务超 40 万大型机构。作为网络安全国家队，奇安信立志为国家构建安全的网络空间，在终端安全、云安全、威胁情报、态势感知等领域的技术研发持续领先。“十四

6、五”规划开局起步，数字化转型全面铺开推动网络安全需求井喷，奇安信将继续为政府与企业等用户提供全面、有效的网络安全解决方案，向成为“全球第一的网络安全公司”的愿景目标不断奋进Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

7、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023原日志安全分析系统输入你的小标题外线框是文字输入区参考线，请在正式稿中删除。文字：根据文章长度选择适合的文字尺寸，以28-24号字为宜。输入你的小标题外线框是文字输入区参考线，请在正式稿中删除。文字：根据文章长度选择适合的文字尺寸，以2

8、8-24号字为宜。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

9、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023架构主要痛点查询性能写入效率 原系统只能通过 SQL LIKE 进行全量扫描和暴力匹配。原系统不支持对文本字段，如 URL、payload 等关键字进行模糊匹配 千亿级数据量的查询耗时近分钟级，当遇到并发查询，性能还会进一步恶化 每天所生产的安全日志数据达到千亿级，且每天新增日志量在不断增长中。原架构系统入库速率逐渐降低，集群写入压力过大、高峰期数据挤压严重、稳定性造成影响 集群多次扩容

10、（3 节点至 13 节点），机器成本已超预期的情况下，写入效率没有得到改善Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

11、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023系统建设目标 海量病毒查杀事件实时写入与存储 基于日志数据 Schema Free 特性支持丰富数据类型的写入与变更实时写入性能 支持对字符串提供模糊查询的能力 能够灵活创建且类型丰富的索引，例如倒排索引以加速筛选过滤数据倒排索引&模糊查询 系统自身的运维简易程度 系统是否具备合适的管控工具简易运维&高效管控工具日志数据价值密度低，

12、但存储的规模很大、存储周期相对较长高性价比存储成本Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

13、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20232架构升级之旅Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

14、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023为什么选择 Apache Doris主要研究探索 OLAP 数据库中擅长日志分析系统的组件，构建一体化日志存储分析平台，赋能网络安全。选择 Doris

15、的理由：1.支持标准 SQL 语法，与 MySQL 高度兼容；2.具备倒排索引、Ngram BloomFilter 索引等检索特性；3.支持聚合、多表 Join、子查询、窗口函数、UDF、物化视图等功能；4.能够在线毫秒级删减字段、按需增减索引、按需更改类型；5.支持 Text、JSON、Array、Map、Variant 等多种数据类型；6.采用 ZSTD 压缩算法优化存储空间占用；Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

16、2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023全新日志安全系统架构

17、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit A

18、sia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023200%690%写入性能提升写入与查询性能提升原系统写入吞吐13台服务器CPU 利用率 30%写入吞吐 30 万/条查询性能提升Apache Doris 写入吞吐3台服务器CPU 利用率 100%写入吞吐 108 万/条原系统查询耗时79 条 SQL 查询语句整体耗时 1757.32sApache Doris 查询耗时79 条 SQL 查询语句整体耗时 253.12sDoris Summit Asia 2

19、023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summi

20、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023存储成本降低40%存储成本降低1.数据压缩比 1:4.31:5.72.数据膨胀率 3-5 倍1 倍ZSTD 是一个优秀的新型压缩算法，使用了智能优化算法，相较于常见的 GZIP 算法，ZSTD 具有更高的压缩率更和更快的压缩解压速度，尤其在处理日志场景时表现非常出色。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

21、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

22、ia 2023运维与管控成本降低Cluser Manager For Apache Doris由飞轮科技免费开放可视化集群管理工具满足日常运维中集群监控、巡检、修改配置、扩缩容、升级等操作。降低登陆机器手动操作的麻烦和误操作风险。可视化日志探索分析 WebUI支持关键词检索、趋势图展示、趋势图拖拽日期范围、明细日志平铺和折叠展示、字段值过滤等交互方便的探索式分析。对于习惯 ELK 日志分析的用户非常友好，契合日志场景探索下钻分析需求。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit As

23、ia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris S

24、ummit Asia 20234Apache Doris 2.0 查询提速实践经验Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

25、 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023SQL查询对比所有查询语句均有明显提升，整体性能查询提升近 7 倍，26 条 SQL 查询语句性能提升 10 倍以上：线上去重分析 79 条 SQL，在同一天 1000 亿条总数据、同样 10 BE 节点集群规模上对比测试查询耗时。8 条 SQL 查询提升 10-20 倍14 条 SQL 查询提升 20-50 倍4 条 SQ

26、L 查询提升 50 倍以上其中差异最高达 88 倍，在原系统执行接近 1 分钟，在 Apache Doris 中仅需不到 1 秒Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

27、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023倒排索引 对关键词查找加速-例如 Q43 提升 88.2 倍 Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202

28、3Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023NGram BloomFi

29、lter 索引 对 LIKE 加速-例如 Q75 提升 44.4 倍 Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023D

30、oris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023TopN 日志明细查询优化-例如 Q22，提升50.3倍 Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

31、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20235规划与展望Doris Summit Asia 2023Doris Summit Asi

32、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

33、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023扩大 JSON 数据类型的应用引入 Variant 可变数据类型支持存储任意结构的 JSON 数据支持字段个数与类型变化更灵活地定义特殊字符，更好地实现半结构数据 Schema Free 分析需求Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit

34、 Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023获取更多社区动态与最佳实践Doris Summit 峰会官网：doris- Doris Summ

35、it 峰会回放：https:/ Doris 官网：doris.apache.orgApache Doris GitHub： Doris 官方平台：Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023