1、腾讯灯塔融合分析引擎设计与实践冯国敬 后台开发工程师|!背景介绍腾讯灯塔、融合分析引擎!#挑战与融合分析引擎的解法!$实践总结!%未来演进方向目录 CONTENT|融合、内核优化、加速背景介绍腾讯灯塔、融合分析引擎!|帮助产品、研发、运营和数据科学团队30分钟内:做出更可信及时的决策，促进用户增长和留存。腾讯灯塔介绍|海量、实时、自定义三角难题三者兼顾需满足条件|腾讯灯塔-融合分析引擎介绍|!#$%&()*+,-!#$%&#()挑战与融合分析引擎解决方案!#|!#$%&()*+,-./0123%4()56789:;+#$%4()?OneFe#ABC;D%EFGHICalciteJ3%KLMNO

2、#OneSQLP%QR4()STUV#Logical-Physical融合分析引擎OneFrontend：多源融合前端|融合分析引擎Backends：融合后端!#$%RDBMSWXY8%Z9-_UYaTb23%ClickHousecdeWXfghijkl$%#$%mnopp()qPrestoImpalarst7connector3%uvwJdbc Connector#xyzSplitProvidersP%|ClickHousecde#DistributedTable Aware-Parallel Local Table%Projection%Aggregation%Predicate Push

3、downJdbcBased ConnectoraTb1CH-ShardcMySQL-Split|融合分析引擎-WLMt7B(ImpalaPrestoClickHouse)!#$%()Profile*F%b23%|EF78$%#$%xWLM(Workload Management)3%xF“AQuery Profilea”EFP%EFmBxEFQuery Options%HintshiTimelineD?hi|融合分析引擎内核优化!#7T-?hi q*+r$%#$%()ZTZ?hi GJ3%=BEFxEFc!SQL#profile$%&(500MRuntimeProfileZT78)*+,IO-

4、Mem-HashJoin-DataSkew.|融合分析引擎-内核优化-Impala!#UY()n($%#$%()bZEF q:4DataRange t%?hiIO br3%t(qPageindex%Zorder%Hillbertr!#$%=aDRnParquet PageIndexTcdImpala IObZEFx(#/-0123456789:-;?|!#PrestojScalabilityCoordinator&?A融合分析引擎-内核优化-Prestohttps:/prestodb.io/blog/2022/04/15/disggregated-coordinatorDisaggregate

5、d Coordinator&#$%(ResourceManager=Ce oZBcd omRcdM3%CoordinatorParse%Validate%PlanRMKeCoordinator HA!#a#/$%&https:/ Group:(4Cluster)*!#$%Resource Fragmentation#e+,F-3%.fg|!#$%?A/T0cd1-23%Tablet Server*E2345q67MasterrQ89:Case#;TabletnZ?ABCMaster7DLeaderEFGH融合分析引擎-内核优化：Kudu&()*+,!-./Heap Profiling/012(

6、)*+,#$%E23TServer#KUDU-29153%I0cdxGC#KUDU-3344-./0#MasterZJK#L$M|融合分析引擎-加速|融合分析引擎-加速：缓存!#m4NO45PQRShTARSQLZTcdUVt7$%#$%WUY qXYZr#mRuOTcdPQ?cdUV3%KO qfhi_sGXYZOr#abCalciteecHISQLdeqfghir%SQLjk%SQLl_P%ZO q?SQLZOr#mmOsSQLnopkOqAlluxiorq?hiHDFSIOrs%Alluxio qHDFStcdOSSDr#m|SQLt7cdDOtuqv?wur|融合分析引擎-加速：BI

7、EngineOracle Database IMBigQuery BI EnginexNcdyz|NMixQuery BI()|融合分析引擎-加速：现代的物化视图gt#Base tableMtNO u SQLk1234.56 WMV(Pre Filtered)W)MV(Pre Joined)W EFMV(Pre Projected&Clustered)(%ZorderHillbert!#HIODSf?hiScan?RFTT$%#$%HISQLqECSErF3%HICalcitexSQLjk78P%(F()78 qx=r78#$%FvHI/DB3%hijk#jkhh“uCP%F()#”(FOx=g实践总结!$|实践总结未来演进方向!%|未来演进方向融合内核优化加速以卓越的性能直接访问数据