1、Apache Doris 在中国邮政储蓄银行金融反欺诈领域的应用探索狄潇然中国邮储银行 软件研发中心 西安分中心技术专家Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

2、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023目录2.应用实践及价值3.未来展望1.业务背景介绍Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Sum

3、mit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20231业务背景介绍Doris Summit Asi

4、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

5、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023企业简介反欺诈平台邮储银行反欺诈平台作为邮储银行智能风控体系的屏障，以科技助力业务发展为理念，前沿技术为支撑，以规则模型策略为基础，打造事前检测、事中阻断、事后分析的全方位一体风险防控体系。实现对全行网络金融、个人金融、信用卡、消费信贷等业务交易的全面风险监控。日均拦截风险交易数万笔，有效阻止客户损失数千万，毫秒间即可完成风险识别。中国邮政储蓄银行拥有近4万个营业网点，服务个人客户超6.5亿户，定位于服务“三农”、城乡居民和中小企业，依

6、托“自营+代理”的独特模式和资源禀赋，致力于为中国经济转型中最具活力的客户群体提供服务，加速向数据驱动、渠道协同、批零联动、运营高效的新零售银行转型。中国邮政储蓄银行拥有优良的资产质量和显著的成长潜力，是中国领先的大型零售银行。Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summ

7、it Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023高时效性运营监测案件溯源分析报表实时性高且精准，能够实时捕获各类风险变化，具备高效、准确、快速响应的能力。支持对交易数据、行为数据、模型策略、风险特征运营情况进行实时监测，一旦发生

8、风险事件，能够进行风险预警和干预。支持对已生成案件进行即席深入追踪溯源分析，包括：欺诈行 为 的 源 头、路 径、涉 及 人员、设备和其他相关信息。实时分析诉求Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

9、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023内部分级建模，风险运营分析涉及多表，数据量大、分析维度不固定且逻辑复杂。多表关联性能优异计划向网点业务人员开放查询权限，查询并发量大、时间集中、案件溯源逻辑复杂。并发查询性能优异功能特点强调实时性，对接入数据实时性有极高的要求，以期迅速捕获风控特征

10、变化。实时写入性能优异某些运营指标的统 计 精 度 要 求高，需要做到端到端的“精准一次消费”语义。事务&数据一致性在监控、灾备、扩缩容、数据迁移等方面有自动化工具，运维成本低，风险小。运维难度低技术愿景Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

11、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20232应用实践及价值Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

12、Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

13、23 架构复杂：多套数据组件，带来高昂的使用成本和极大的运维、管控压力。性能不足：每个场景一个组件，单一组件难以满足业务对数据分析效率的需求，时效性差。大数据分析领域现状 对比分析了 ClickHouse、GreenPlum、Cassandra、Kylin 等多个列式存储组件。着重对 Doris 和 ClickHouse 进行对比分析。Doris VS ClickHouse技术选型Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20

14、23Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202389个测试案例11条修改

15、意见，7次修改迭代10个文件，156页，5W多字评测过程5轮安装部署Apache Doris在实时写入性能方面表现优异，现有硬件资源下，实时写入性能是Clickhouse的6倍，且支持事务，支持“精准一次消费”语义。评测结论在单表查询场景下，ClickHouse的表现相对更优，适合查询模式固化的宽表分析；Doris的单表查询性能良好，能够满足当前业务场景中单表查询性能要求。Apache Doris在多表Join、复杂聚合及高并发场景下的查询性能具有明显优势。Apache Doris具备完善的高可用方案，可自动完成切换，无须人工干预。TPC-H 数据集SSB 数据集技术选型Doris Summi

16、t Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dor

17、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Pg维表2维表1消息中间件准实时数据2准实时数据1准实时数据4准实时数据3业务数据离线文件文件2文件1数据采集Online-ProcessPg-sourcePg-sourceKafka-source多流JoinETLDoris-sinkETL&Java-UDFDoris-sinkFlink sql&connectorOffline-Process数据清洗数据转换数据加工风控运营监测模型策略概况数据应用即席分析风险特征案件溯源T

18、able1Table2Topic1Topic2库表同步数据打宽数据清洗批量加工Doris-sinkKafka-source数据建模DIMODS:实时、离线接入的贴源数据分钟粒度实时聚合HiveKafkaPg联邦分析原始数据DWD:按业务维度分解数据DWM:分钟粒度实时聚合数据DWS:基于DWM层上卷按小时/天粒度聚合数据Info1Info2Info3动态分区自动分桶Sum聚合Bitmap去重过期策略前缀索引Duplicate模型小时粒度天粒度离线聚合实时聚合Rollup物化视图Aggregate模型Rollup高度聚合解决方案（脱敏）Doris Summit Asia 2023Doris Su

19、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023

20、Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 202303工单标记04运营预警01监控报表Doris+Flink流计算引擎联合，完成基于规则、模型设备、地址等维度报表的实时统计。02即席分析基于近一年交易记录，提供包含10多种复杂条件的实时统计分析。支持工单信息与交易信息进行关联分析，快速定位阻断详情。实 时 接 入 系 统 告 警 信息，作业运行数据、模型策略运行数据。场景应用Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dori

21、s Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia

22、2023Doris Summit Asia 20231.支持对1000万+客户、3万+柜员、1万+支行网点、1千+产品的在线统计分析。2.支持1万+笔工单的数据追踪、详情分析。3.支持动态检测近百种不同类别的告警规则，实现系统级别预警提醒。业务量1.核心报表数据实时性从1-2天延迟骤减至5s内。2.80%即席分析可在2s内返回结果，95%的即席分析可在5s内返回结果。3.受益于Apache Doris极致的存储压缩比，存储成本降低70%。业务效果应用价值Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

23、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2

24、023Doris Summit Asia 2023告警延迟告警有效率近两月，触发规则模型运营告警100次955s内超%+应用价值Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asi

25、a 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 20233未来展望Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Dor

26、is Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023v1.0.0v1.1.0日志检索v1.1.1高并发点查v1.1.52.0 升级

27、现状使用ELK、Kafka、Flink、Hive等组件进行日志采集，涉及组件较多，不易管理，开发运维成本高，且占用了较多服务器硬件资源。优化统一采用Kafka+Doris的轻量级日志采集框架，实现日志统一管控，以SQL的方式便捷查询，降低开发运维难度，在不影响原业务基础上节省服务器资源。现状Doris基于列式存储引擎构建，对于大宽表高并发查询整行数据的场景下，会导致CPU开销较高，进而影响其他正常业务的资源使用。优化实时数仓在面向不同分行、支行时，能够承受住高并发下行式数据的检索分析，提供大数量级下明细数据的统计分析。当前版本无法支持以下业务场景（1）在Unique模型中使用数组类型进行数据存

28、储。（2）日志关键字模糊查询、等值查询和范围查询场景。未来展望Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris

29、 Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023获取更多社区动态与最佳实践Doris Summit 峰会官网：doris- Doris Summit 峰会回放：https:/ Doris 官网：doris.apache.orgApache Doris GitHub： Doris 官方平台：Doris Summit Asia 2023Doris Summit Asia 2023Doris Su

30、mmit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023Doris Summit Asia 2023