1、TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 署名：封仲淹(Longda)职称：资深专家 Large-Scale Stream Processing inside AlibabaLarge-Scale Stream Processing inside Alibaba TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 自我介绍 自我介绍 JStorm/Storm evangelist in China Our duties:Application Develop

2、ment JStorm Platform Evolving JStorm/Storm Support Cluster Management Community Commitment TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 目录 目录 User Scenario Architecture Enhancements Performance Stability Features TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM JStorm Milestone

3、 JStorm Milestone 2013/4/30 0.7.1 Java core Scheduler 1.0 2013/12/30 0.9.0 Nimbus HA 1.0 Backpressure 1.0 Scheduler 2.0 2015/10/27 0.9.8.1 Koala 2.0 Scheduler 4.0 Backpressure 2.0 Topology Master 2015/11/12 2.1.0 Donate to apache Support Storm 0.10 binary Metrics Redesign Koala 3.0 Release 25 Jstorm

4、 version from 2013/4/30 0.7.1/0.9.0/0.9.1/0.9.2/0.9.3/0.9.3.1/0.9.4/0.9.4.1/0.9.5/0.9.5.1/0.9.6/0.9.6.1/0.9.6.2/0.9.6.3/0.9.7/0.9.7.1/0.9.7.2/0.9.8/0.9.8.1/0.9.8.2/2.0.4/2.1.0/2.1.1/2.2.0/2.2.12014/10/11 0.9.6.3 Koala 1.0 Scheduler 3.0 User Defined Log User Defined Metrics 1.0 2016/5/31 2.2.0 Improv

5、e performance Exactly Once Sliding Window Shade Dependency Backpressure 3.0 TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 当前 当前 Everywhere 4000+machines Standalone JStorm-on-yarn JStorm-on-docker 1500+applications,2000+topologies 2+PB every day TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD

6、TRADE PLATFORM 应用场景 应用场景 Fraud Detection:Nut/Velocity Auditing:Alimama Ads auditing,Alimama P4P auditing,AMG repository auditing,Statistics:Eagleeye,AE pv?uv statistics,BI statistics Monitoring:Tlog,Rds-monitor,Oceanbase-monitor,Cainiao Raidar,QuaMonitor,YunOSMonitor,Data Transport:RDS-log-sync,Unif

7、y-Log Realtime Recommendation:Alipay 1315,Alipay Hyperloop,Tpp Scheduling Apps:AE mail realtime analyzer,1688 Wangxiaobao TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM JStorm 生态 JStorm 生态 TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 架构 架构 TLog Kepler NUT ARM RDS Trip YunOS C

8、ainiao Porter(Koala)SQL engine Plugin（Kafka/Meta/TT/Mysql/HDFS/Hbase/）CEP Plugin Trident Exactly Once Extension API Raw Storm API JStorm Core Local Cluster Standalone,YARN,Fuxi Cloud(Docker,ECS,.)TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 0 2 4 6 8 10 12 2 worker,1 spout,2 split,2 cou

9、nt 10 worker,5 spout,20 split,20 count 10 worker,10 spout,40 split,40 count 20 worker,10 spout,80 split,80 count 40 worker,10 spout,160 split,160 count 80 worker，10 spout，320 split，320 count Million JStorm Storm 性能 性能 HW:32 CPU core/128G Storm version:1.0.2 Test:Storm FastWordCount JStorm version:2.

10、2.0 Worker:6G RAMTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM Pipeline Application Deploy Core 性能 性能 Task Pipeline Tuning UserDefined scheduler Improve efficiency Improve parallel Shuffle Resolve bottleneck Batch DesignSerialization TuningSeparate spout send/recvMetrics TuningRouting Op

11、timizationOverhead ReductionTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM Isolation High Availability Monitoring Degradation Multi-Cluster Deployment 稳定性 稳定性 Backpressure UserDefined degrade Message lifecycle Userdefined metrics Userdefined alarm Single failure:nimbusHA Node self check:c

12、heckhealth Huge deploy:topology-master Across cluster:jstorm-on-docker/standalone In one cluster:jstorm-on-yarn/cgroup Clusters in different city:hot backup/cold backup Clusters in one City TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 监控系统 监控系统 TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD

13、 TRADE PLATFORM 自定义监控 自定义监控 Improving performance:Multiple Layer Metrics Design Automatically aggregate task levels metric to component level,then aggregate into topology levelShare common metricsTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 自定义监控 自定义监控 TECHNOLOGIES CONSTRUCTING ELECTRON

14、IC WORLD TRADE PLATFORM NimbusTopology MasterTaskZK ClusterTopologyTaskTaskTopology MasterTaskTaskTaskTopology MasterTaskTopologyTaskTaskTopology MasterMetricsTask heartbeatControl event大规模部署 大规模部署 TopologyTopology Master slaveTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM JStorm Backpres

15、sure 3.0反压 反压 Netty clientNetty cacheSending taskNetty serverTask cacheReceive taskIn one workerTask cacheReceive taskExceed ThresholdTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 其他强化功能 其他强化功能 User Side Functionalities User-Defined Scheduler User-Defined Resource Usage Advanced Logging

16、Mechanism More Fine-Grained Metrics Graceful Shutdown Dynamic Rescale/Reload/Restart Receive Command Quick Application Upgrade TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM Exactly OnceExactly Once Chandy-Lamport Algorithm Barrier Stream Aligning TECHNOLOGIES CONSTRUCTING ELECTRONIC WORL

17、D TRADE PLATFORM Spout Stateless bolt State bolt Ending bolt Topology Master Cache dbHdfs/hbaseNormal batchControllerNexttupleCommitCache dbExecuteExecuteCommitExecuteExecuteCommit offsetCheckpoint barrierCheckpoint barrierJStorm checkpoint(optional)Jstorm checkpointCheckpoint barrierCheckpoint barr

18、ierCheckpoint and clear old checkpointExactly OnceExactly Once TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM Exactly OnceExactly Once Spout Stateless bolt State bolt Ending bolt Topology Master Cache dbHdfs/hbaseControllerNexttupleRollbackCache dbExecuteExecuteRollbackExecuteExecuteRollb

19、ack barrierRollback barrierRollback barrierRollback barrierRead last success checkpoint and clean useless checkpoint startrollbackRead checkpoint(optional)rollbackTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM FutureFuture Apache Beam A more powerful high level programming framework Easie

20、r to learn,to debug Provide higher throughput TECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 开源现状 开源现状 50+CompaniesTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM 总结 总结 Why Start JStorm?Move faster Whats the difference between JStorm and Storm?JStorm means Stream Process Ecosystem JStorm is Enterprise Java Storm More faster More stable More featureTECHNOLOGIES CONSTRUCTING ELECTRONIC WORLD TRADE PLATFORM zhongyan.fengalibaba-?www.jstorm.io