1、Distributed Agile and DevOpsPractitioner approach Avanish TiwariAvanish TiwariIndustry Principal and Head of DeliveryIndustry Principal and Head of DeliveryInfosys Technologies(China)Company LimitedOrganizational Context 组织背景3复杂的环境更迅速的变化更多的工作复用和改造更快、更优、更智能Time allocation for operation team运维团队的时间分配4

2、Source:Study from Deepak Patil Microsoft Global Foundation Services in 2006,via James HamiltonDeployment management and fix deployment related incident take up nearly half of their time(47%)in typical large and medium-sized operations teams运维团队有将近一半（47%）的时间花在了与部署有关的工作中：1.日常新增功能的部署2.修复与部署有关的问题DevOps

3、Philosophy centers around“Enterprise Agility”DevOps注重于“企业敏捷性”Speed to ValueChange ImaginedPrioritizeChange DeliveredAccelerateAdoptShift leftLean&agile cultureProblem-findingEmpathy-based stakeholder engagementInsights-based learningContinuous feedbackContinuous deliveryTime to ValueCustomer Centric

4、ityCost EffectivenessService Reliability and QualityIntegratedtoolingOrganizational Readiness&Maturity Assessment a glimpse组织准备状态和成熟度评估Key Activities Define DevOps goals Define scope Identify key stakeholders Conduct a workshop/surveys with identified stakeholders Understand the current initiatives,

5、priorities,challenges andexpectations IT value stream analysis Portfolio analysis,DevOpsreadiness and maturity assessment,analysis and gap report DevOps technology assessment and tooling gap analysis Create assessment report,playback the findings with the stakeholders and incorporate the feedbackDev

6、Ops readiness and maturity assessmentImportant DevOps Metrics DevOps 重要指标Focus AreaMetricFormulaProposed UsageRelease frequencyDeploys/DayTotal#of deployments/#of days To understand increased collaboration&keep a track of time based improvementReleases/Week Total#of releases/week(by environments)Imp

7、rovement in production release agility over the period of timeAgilityChange Lead TimeTime required to release a change in production from the time the development begins on the change requestTo know the trend in lead time&help remove any bottlenecks if a degradation is observed QualityCode/Test Cove

8、rage%of code covered through the test cycle,(Unit test)%functionality covered through test cycle(regression test)To know the testing effectiveness through the developed codeRelease Efficiency/repeatability%of successful releases over a period of timeHelps measure effectiveness of CD setup&observe tr

9、end over a period of timeProduction Defect MatrixAverage#of defects released into production Helps measure effectiveness of quality control through continuous delivery setupLead time provisioningTime taken for provisioningTo know the trend in lead time&help remove any bottlenecks if a degradation is

10、 observed PerformanceAvailability%availability of serversDetermine stability of environments across enterpriseOutage Outage per day/week/monthTo measure availability against SLACapacity%capacity availability&utilization Gauge adequate capacity planning&useReliabilityMean Time to RecoverTime required

11、 to recover from outages Improvement in MTTR over the timeEfficiencyCost or effort per releaseInitial effort/cost per release compared to post DevOpsTo understand how efficient release process is IllustrativeWhat is Agile?What should be the investment?How to implement agile in our business/technolog

12、y context?How to enable teams and business stakeholders?When to choose agile for a project?How to replicate success of small projects to large size projects?Why certain agile projects fail?Why we do not get desired agile benefits?How to perform multi-location execution?How can I improve on my Agile

13、adoption?How Agile and waterfall co-exist in a portfolio?How to standardize the process across organization?While IT uses Agile,we really dont get frequent production releasesHow to measure/report agile projects?How can I scale up Agile adoption?Entrant,practitioner or expert,organizations still hav

14、e concerns to be addressed新手、从业者或专家、组织仍有待解决的问题Deploy to multipleenvironments 多环境部署Gated deployments 原型部署Artefact repository based deployments 基于版本控制系统Container based deployments 容器部署Service testing 服务测试Performance testing 性能测试Virtualization 测试结果可视化Functional testing 功能测试Security testing 安全测试CONTINUO

15、US DEPLOYMENT持续部署CONTINUOUS TESTING持续测试完善的DevOps管理平台Automated code reviews,Unit testing,Coverage,Obfuscation,Security scan 自动化代码测试Auto setup of Open Source tools 开源工具的自动化安装CONTINUOUS INTEGRATION持续集成预测和挖掘的能力Predication&Insights9DevOps Platform DevOps平台的能力Increased Adaptability to ChangeIncreased Spee

16、dImproved Visibility Quicker DecisionsFail Early,Fail CheapReduced WastageDrives Innovation and CompetitionCutting CornersFitting more in less budgetJust replacement for WaterfallEasy to implement:No need to train or coachOnly coding and nothing elseEscape from processesBut Agile does not guarantee

17、success.It only increases the likelihood of success 敏捷不保证成功。它只增加成功的可能性Agile is notAgile deliversAgile has gained wide acceptance for its showcased benefits and quick success with small projects敏捷已经被广泛接受，因为它展示了小项目的快速成功Traditional Approach towards Setting up CI-CD需要从传统的工作方式走向持续集成-持续交付Create workflowsS

18、et UpTools orchestrationMaintenanceCI-CDComplexities in integration of various tools整合多种工具的复杂度Lack of tooling and skills 工具和技能缺乏Effort intensive需要巨大的工作量来测试，部署和交付Process&compliance overheads-approvals and role based governance审批管控，流程和协作的管理工作量流程Process人员People技术Technology11Redundant tooling ecosystem冗

19、余的工具和功能，割裂的支撑系统Reduced visibility&governance缺乏可视化和管控Delay in Time to Value set up,integrate&adopt交付的延迟Significant maintenance overheads巨大的维护和管理成本Disparate CI-CD orchestrations完全不同的集成和上线节奏Challenges get amplified in a larger IT landscape在多团队的组织中，运营挑战会成倍上升Multiple teams re-inventing the wheel多团队相互协同影响

20、Problem areas 发生问题的领域Set UpTools orchestrationMaintenanceCI-CDCreate workflowsSet UpTools orchestrationMaintenanceCI-CDCreate workflowsSet UpTools orchestrationMaintenanceCI-CDCreate workflowsMultiple Application teams 12Agile is more relevant today for dynamic market conditions but its time to chan

21、ge the agile way今天，敏捷与动态市场情况更加相关，但现在是时候改变敏捷方式了Agile is the way for IT to remain relevant to BusinessChallenging market conditions make Business demand more from ITReduced time-to-market Low upfront investmentEarly realization of business valueMonitoring delivered valueagainst fundingIncreased Adapta

22、bility to ChangeIncreased SpeedImproved VisibilityFail Early,Fail CheapReduced WastageImproved QualityBut,co-located,small team focused agile is not sufficientLosing cost advantage and talent availability Inability to rapidly scale up agile execution Not able to generate business value rapidlyChangi

23、ng Agile with changing timesCo-located ToDistributedProjects To EnterpriseIT goalsToBusiness valueChanging Agile and its key considerations改变敏捷方式及其关键点 Communication Collaboration Maintain agile spirit Scalability Faster adoption Organization challenges Reduced cycle time Time to Market Productivity

24、gain Low cost Improved quality Customer satisfactionChanging Agile with changing timesCo-located ToDistributedProjects To EnterpriseIT goalsToBusiness value1231.Changing Agile from Local to Global 敏捷从本土转向全球Traditional Agile High cost Small teams with scalability challenge Rapid development of featur

25、es by group of smart developers in random fashionGlobal Agile Global Agile with state of art infrastructure for global communication and collaboration Distributed agile practices Global Agile:a structured,risk controlled,measurable delivery methodology.Expertise:agile trained talent pool across glob

26、eHow do we do it?Robust Process and IPsCollaboration Talent Enablement Virtual ScrumBoot CampAgile Scrum Process TrainingAgile Mindset TrainingAgile AwarenessScrum Master and Team View152.Shift focus from rapid development to quicker business value realization关注点从快速开发转变为更快的商业价值实现16Traditional Agile

27、Agile is a project development methodologyGlobal Agile Realization of business value is a key to agile success.Rapidly developing code which is released less frequently leads to delay in realizing business value.Moving from agile development to agile delivery is a solution.Building automated continu

28、ous delivery pipelines leading to increased release frequency ensures frequent business value delivery.How do we do it?Building Engineering Ecosystems for continuous deliveryalliances beneficial to our customers 与客户共赢Progression from Agile development Agile Delivery DevOps敏捷开发 敏捷交付 DevOps3.Change Ag

29、ile from projects to enterprise-wide:structured,repeatable 敏捷从项目延伸到到企业范围：采用结构化的、可重复的方式17Traditional Agile Consultants add theory but execution is a different reality.Scaling agile is doing more and more agile projectsGlobal Agile Theory to practical Consulting capabilities coupled with large scale p

30、roven delivery capability ensures end to end responsibility of clients agile journey Structured scaling from projects to portfolios to enterprise coupled with organization change management How do we do it?Largest pool of consultants and coaches providing servicesEnterprise Scalability FrameworkAgil

31、e Maturity Assessment FrameworkAgile DeciderScrum Adoption FrameworkRisk Management Framework BVACXO dashboardITPA-IT process enablerAgile Readiness/Maturity AssessmentEnterprise Agile Transformation Rapid Agile Adoption Agile CoachingAgile TrainingTools and products evaluationAgileCoaches and consu

32、ltants.and moreAgile journey by adopting Global Agile 全球敏捷之旅Global Agile Ecosystem全球敏捷生态系统Coaching&Talent DevelopmentProcessRole basedTechnologyToolsSoft skillsMindsetCultureCapabilityPracticesToolsGovernanceGlobal AgileCase Reference 案例参考Customer Delivery ModelUSA-HeadquartersOffshore Development C

33、enterThe program follows a Global Scrum Delivery model from 2 different geographical locations USA Customer Onsite locationOffshore Development Center in IndiaContextThe Engagement is deals with Partner eco-system of the Large Hi-Tech customer.This engagement has few applications that offers several

34、 tools and platforms to its partners.Provides enrollment,Re-enrollment and program/benefits access for ISV partners;manages annual enrollment+profile,Serves as Partner Data Master.ScopeDelivering key business functionalitiesFeature Development-EnhancementsSustain EngineeringTestingBuild EngineeringX

35、-Geo DeliveryCost Savings and Time Advantages(Follow-the-Sun)were the key drivers for adopting a X-Geo delivery by the customer.10 feature crews and 2 Sustain crews with more than 90%of personnel presence.Program Context Global Agile delivery in scale规模化全球敏捷交付Key Challenges 关键挑战Huge pressure due to

36、a recent failure of another vendor team to create a new platform replacing the current.Low CSAT in using current platform.High volume of support ticketsBacklog from business pending to be delivered for partners through features;Acceptance criteria not defined well/not completeUrgent demand to ramp u

37、p new teams in IT(India/USA)Accumulated bugs,technical debts(automation,performance)in the current platform.CICD was not implemented and deployments took longer than expectedAgile transformation of the new teams.Transparent metrics to measure output through VSO adoption.Slower time to market,with hu

38、ge number of integration points and dependencies to be regressed every release.Agile teams working across geographies to align to a solution.Low morale due to aggressive timelinesTraditional culture on understanding on roles and responsibilities of a SWE.Client problem-Need for transformed agile tea

39、ms in scale.客户需要规模化的敏捷团队Client expectations客户期望Increased CSATReduced bug trendImproved demand management and traceability from portfolio through to Agile delivery teams Improved throughput and productivity from the teams.Faster time to market22SAFe Principles and Agile practicesOrg wide approval fro

40、m customer to adopt SAFe including business stakeholders with commitment to business results.Created optimally sized,cross-functional feature crews(Scrum Master,Product Owner,SWEs).Defined roles,artifacts,and agile team performance metricsIntroduced Portfolio,Program,and Team levels making up the Ag

41、ile Release TrainIntroduced backlogs with Epics,Features and stories in VSO and maintained hygiene of updates.Introduced Normalized Story points and a common estimation template across teams.Set up cadence(2 week sprints)to synchronize and align with the crews.Built an effective“Sprint ahead”practic

42、e to ascertain delivery of sprint commitment.At the program level had a 12 week planning increment event.Created parallel teams to address automation(Build,Deploy,Regression and Load tests).Created a systems team to help set up CI-CD to gate check-ins,deployments and code promotions and manage envir

43、onments.1Agile Teams structure(90%team located in offshore)敏捷团队结构（90%为离岸成员）Feature Crew Lead/Scrum Master(1)SWE(7)Story Points FeaturesBugsFeature Crew 1(8)Feature Crew Lead/Scrum Master(1)SWE(7)Story Points FeaturesBugsFeature Crew 2(8)Feature Crew Lead/Scrum Master(1)SWE(7)Story Points FeaturesBug

44、sFeature Crew 7(8)Feature Crew Lead/Scrum Master(1)SWE(7)Story Points FeaturesBugsFeature Crew 8(8)Stabilization Crew(20)(Note:Functions can be optimized into feature crews as they get more self-organized)Regression(5)Release Manager(RTE)Product Owners(8)Sustain Crew(10)Build(5)Perf Testing(5)Svc En

45、gineering(5)Bug Triage Support ProcessRoot cause,bug fixes,Improve Live site monitoring through Power BI Dashboards.Data workaroundsAdhoc business change/data requestsModularize Regression test suitesClear Automation debts(Build,Deploy,Regression,Perf tests)Enable CI-CD Monitor DeploymentsEnvironmen

46、t Support and MaintenanceMonitor System health,system performance and availability.Onsite 现场Offshore 离岸Zero-defect Feature DevelopmentUnit tests,Acceptance TestsFixes defects from FortifyFixes defects from regression suite.Fixes defects from Load testsIntegrate Acceptance tests into regression suite

47、Self report on Sprint performance metricsAccountable for project success or failure.Product Owner manages the Product Backlogs,stories,acceptance criteria.Owns what is desired and why is desiredInterfaces with Release manager,SMEs,Business stakeholders to engage with scrum team.Release calendarFacil

48、itate Program Increment workshopsTracks and mitigates release level issues/risks.Program level status reportWorkshop GoalQ2 program Increment PlanAlignmentTypical Quarterly Program Increment planning workshop典型的季度产品增量规划季度研讨会Sprint 1Iteration 1Sprint GoalsIteration 2Sprint GoalsUAT&DeployDemo&RetroSp

49、rint 2Sprint 3Sprint 4Iteration 1Sprint GoalsIteration 2Sprint GoalsDemo&RetroSprint 5Sprint 6Sprint 7EpicsEpic,User StoriesUAT&DeployIteration 1Sprint GoalsIteration 2Sprint GoalsDemo&RetroUAT&DeployFeature PM manages the Product Backlogs.Owns what is desired and why is desiredSprint aheadSprint ah

50、eadSprint aheadEstimateEstimateEstimateCode completion and unit testingCode review and reworkCode check-inAutomation test scriptingEstimate Sprint BacklogDefine Sprint scopeCommit to sprintRegression Pass,Demo&RetroRegression pass,Demo&RetroRegression pass,Demo&RetroSprint aheadEstimateSprint aheadE

51、stimateSprint aheadEstimateNext Program IncrementRelease 1Release 2Release 3Scrum Master:Keeper of scrum process,facilitatorTeam:Owns how and how quickly work is deliveredProgram IncrementRelease 2Release 3Technology,tools andFrameworks1.Visual Studio Online2.ASP.NET MVC/WEB API3.JQUERY4.Fortify sca

52、ns5.Style Cop4.MS unit (Unit tests)5.Selenium web driver(Feature tests)6.PowerShell scripting1.Release manager2.App Insights(MSIT data link)3.SQL 2014,SSIS1.Azure Web apps2.SQL Azure3.Azure active directoryWeekly status reporting depicting all the below pictorially to all business stake holders ona.

53、Release/Sprint calendar with monthly release cadence b.Progress of the features,risk/issues.e.Weekly support tickets trend old/new,application availability,highlight and lowlights26PlatformImprovementsImproved Telemetry needs that helped in faster resolution of bugsAddress environment gaps to be on

54、par with production for early visibility of defects.Addressed Regression tests,Loads test automation and Build and deployment automation.Continuous Integration using Release Manager to boost release confidenceGated Check-ins with unit tests Daily Build with integrated BVTs,automated security scans.C

55、ontinuous deployment pipeline gated with the right categories of tests(Feature/Regression)Built POWERBI dashboards with comprehensive set of production scenarios for live site monitoring to address problems proactively to avoid negative CSAT.Implemented few self-serve tools to fix data bugs so susta

56、in crew could focus on business features and also improve CSAT.2自管理成员的敏捷性能指标示例566702468S32S33S34S35Story points delivered per person/SprintVelocity TrendTrack the team effort completed for each sprint.Estimate the backlog effort that team can handle in future sprints if the team composition and spri

57、nt duration stay constant.Sprint Burn down chartTrack remaining work in a sprint backlog,understand how quickly team has completed tasks,and predict when the team will achieve the goal or goals of the sprint.Productivity TrendThis will help the Scrum Master to track the Productivity improvement with

58、 each iteration for each member in the team Bug TrendTrack the teams progress toward resolving and closing bugs by viewing the Bug Status report.This report shows the cumulative bug count based on the bug state,priority,and severityReduced Bug TrendReduced bugs from 300 to 90 through Telemetry,Power

59、 BI reportsEarly visibility to defects with improved test suites as a service for feature,regression and load tests.Followed TDD where Acceptance tests were automated using GHERKIN and Specflow.Regular Cadence with Service Engg and Helpdesk teams to control new tickets.Improved ThroughputCrews veloc

60、ity improved from sprint to sprint;Productivity Improvement from 750 SP to 1400 SP in 2 quarters with same team sizeSprint Ahead Feature PM has realized in improved Sprinting Ahead,saving 2 dev days per Sprint team.Faster Time to MarketAutomated the build and deployment process by incorporating CI/C

61、D builds(80%effort reduction in deployment cycle in various environments,60%reduction of downtime)Enhanced the regression automation suite and performance suite to reduce the regression effort.(40%reduction in Regression Testing cycle time)Reduced UAT timeline to the bare minimum by ensuring zero es

62、caped defects.Increased CSATDemo/Review the features with Business groups for each release has enhanced the partner experience Enabling Service Engg,Helpdesk on new features for smooth support and quality experience to partners.Self serve tools to enable RSCs to resolve data issues with Quick TAT(without engineering team interference)Stabilizing the eco-system with more modern engineering practices.Questions?Thank You