1、历时三年的超大型软件系统DevOps之路DevOps是一种从开发到运维的快速的相互反馈的文化，并没有具体的标准模式，但从自动化入手，最容易取得成效。本次议题主要分享实战经验，可以快速应用到新的产品研发中去。Session 1:DevOps 相关概念00:00 00:15Session 2:搭建DevOps 生产线00:15 02:30Session 3:问答交流02:30 03:0003:00 闭馆:继续问答交流微信/邮件：继续问答交流Session1:DevOps Pipeline?2019/9/234传统瀑布模型 基线绿线为时间轴红线为基线化的文档或文件交付的产品交付的产品系统需求需求软件

2、需求软件需求设计规格说明设计规格说明需求分析软 件 设 计编码/单元测试软件集成测试验 收 测 试代代 码码待交付的产品待交付的产品维 护2019/9/235产品研发管理文档管理产品信息变更管理需求管理代码版本缺陷管理客户反馈测试用例Branch model:Git flow$git flowusage:git flow initInitialize a new git repofeature Manage your feature branches.bugfix Manage your bugfix branches.release Manage your release branches.

3、hotfix Manage your hotfix branches.support Manage your support branches.version Shows version information.config Manage your git-flow configuration.log Show log deviating from base branch.Git commands wrapper:to implement the processes and integrate other tools1.Change jira ticket status2.Tag naming

4、 convention3.Branch naming convention4.Auto Rebase.Waterfall way of workingAgile&Lean way of workingProduct readyfor deliveryTrouble report inflowTeam learningWATERFALL VS AGILE&Lean WAYS OF WORKINGSystem test progressTRANSPARENCY TASK BOARD DevOps is a set of software development practices that com

5、bine software development(Dev)and information-technology operations(Ops)to shorten the systems-development life cycle while delivering features,fixes,and updates frequently in close alignment with business objectives.DevOps DefinitionDevOps Definition Academics and practitioners have not developed a

6、 unique definition for the term DevOps.“Academic defining DevOps as a set of practices intended to reduce the time between committing a change to a system and the change being placed into normal production,while ensuring high quality.ToolchainsAs DevOps is intended to be a cross-functional mode of w

7、orking,those that practice the methodology use different sets of toolsreferred to as toolchainsrather than a single one.18These toolchains are expected to fit into one or more of the following categories,reflective of key aspects of the development and delivery process:19unreliable source?20unreliab

8、le source?Coding code development and review,source code management tools,code mergingBuilding continuous integration tools,build statusTesting continuous testing tools that provide quick and timely feedback on business risksPackaging artifact repository,application pre-deployment stagingReleasing c

9、hange management,release approvals,release automationConfiguring infrastructure configuration and management,infrastructure as code toolsMonitoring applications performance monitoring,end-user experienceSome categories are more essential in a DevOps toolchain than others;especially continuous integr

10、ation(e.g.Jenkins,Gitlab,Bitbucket pipelines)and infrastructure as code(e.g.,Terraform,Ansible,Puppet).21unreliable source?22unreliable source?Forsgren et al.found that IT performance is strongly correlated with DevOps practices like source code management and continuous delivery.13Relationship to A

11、gile Agile and DevOps serve complementary roles:several standard DevOps practices such as automated build and test,continuous integration,and continuous delivery originated in the Agile world,which dates(informally)to the 1990s,and formally to 2001.23Agile can be viewed as addressing communication g

12、aps between customers and developers,while DevOps addresses gaps between developers and IT operations/infrastructure.24Also,DevOps has focus on the deployment of developed software,whether it is developed via Agile or other methodologies.23.Relationship to Continuous delivery Continuous delivery and

13、 DevOps have common goals and are often used in conjunction,but there are subtle differences.2728 While continuous delivery is focused on automating the processes in software delivery,DevOps also focuses on the organizational change to support great collaboration between the many functions involved.

14、27 DevOps and continuous delivery share a common background in agile methods and lean thinking:small and frequent changes with focused value to the end customer.29unreliable source?Lean management and continuous delivery are fundamental to delivering value faster,in a sustainable way.14Continuous de

15、livery focuses on making sure the software is always in a releasable state throughout its lifecycle.13This makes it cheaper and less risky to deliver the software.13 Improved collaboration and communication both between and within organizational teams can help achieve faster time to market,with redu

16、ced risks.3031 move faster and be more agile without sacrificing security or reliability.DevOps is a cultural perspective on how everyone should be engaged in working the right way.DevOps is a culture,not a role!The whole company needs to be doing DevOps for it to work.DevOps is about continual lear

17、ning and improvement rather than an end state.DevOps:BenefitsDevOps:BenefitsDevOps:AutomationAgile-CI-CD-DevOpsAgile-CI-CD-DevOpsAgile-CI-CD-DevOpsDev+OpsAgile vs.DevOps怎么做到的搭建前，制定目标与量化指标：易于维护过程透明，白盒化开发人员易于使用简单化架构易于扩充与修改配置便于支持与维护搭建过程中，采用敏捷方式，分期迭代部署出现错误时可以及时回溯到历史版本不同任务之间要有清晰的接口可移植性促进而不是阻碍开发尽量使用工具的原生命

18、令日志记录便于分析改进Session 2:DevOps PipelineDevOps toolchainDevOps toolchainDevOps toolchainBitBucket Integrations Bitbucket is more than just Git code management.Bitbucket gives teams one place to plan projects,collaborate on code,test,and deploy.https:/bitbucket.org/product/Jira SoftwareImprove efficiency

19、 by connecting branches,commits,and pull requests to Jira Software issues.Trigger webhooksIntegrate easily with systems you already use in your day-to-day using Bitbucket webhooks.git Git Large File SupportSourcetree supports Git LFS,allowing teams to track large assets in one single place.Git-flow

20、out of the boxSmart branching with Sourcetree and Git-flow,keeping repositories clean and development efficient.SubmodulesSubmodules make life easier when managing projects,their dependencies,and other project groupings.Local commit searchSearch for commits,file changes,and branches right within Sou

21、rcetree.Interactive rebaseGet clean and clear commits with Sourcetrees interactive rebase tool.Remote repository managerSourcetree allows you to search for and clone remote repositories within its simple user interface.Pipelines We accelerate release cycles from once per month to twice per week,allo

22、wing QA and product teams to test early and get customer feedback quickly“Manage your entire workflow in one toolEmpower your team with a fully visible pipelineJira The#1 software development tool used by agile teams The best software teams ship early and often.Choose a workflow,or make your own Int

23、egrate with the tools you already useJenkins 构建伟大，无所不能 Jenkins是开源CI&CD软件领导者，提供超过1000个插件来支持构建、部署、自动化，满足任何项目的需要。https:/jenkins.io/zh/doc/Jira SW WorkflowTransitionTriggersStart progress(To Do In Progress)Branch created(Bitbucket Server)Commit created(Bitbucket Server)Start review(In Progress In Review

24、)Pull request created(Bitbucket Server)Pull request reopened(Bitbucket Server)Review started(Crucible)Restart progress(In Review In Progress)Pull request declined(Bitbucket Server)Review rejected(Crucible)Review abandoned(Crucible)Done(In Review Done)Pull request merged(Bitbucket Server)Review close

25、d(Crucible)One Jenkinsfile on all branches,with default values.Code static analysis:sonar/coverity软件模块化、独立化1.dependencies from artifactory server2.try not use local file system快速编译是最核心部分，如何增量编译和并行编译自动完成绝大部分的评审，尽量减少人工评审工作量自动完成持续集成流水化作业：Jenkins从simple job到pipeline到JenkinsFile，持续集成环境也要和代码一起进行版本管理使用虚拟机和

26、容器来提升效率如何提高构建速度：Gradle,Electrical Accelerator,ccache,RAID0,Makefile测试也分布式完成构建结果的管理和提升以Docker Images的方式发布持续交付中自动更新产品数据库ALL in Git version control:Server Env Configurations Jenkins Job Configurations Source CodeJenkins Job Builder or Jenkins DSL Plugin:Jenkins Job Configurations对比http:/ JenkinsProduct

27、ion Jenkins的的界面修改界面修改JobJob配置配置Server Env ConfigurationsJenkins Job ConfigurationsJenkins Job Inside ScriptsTwo Instances:StagingProductionServer Env Configurations in AnsibleJenkins Job Configurations into DSL script用JobDSL脚本生成Jenkins JobIn Jenkins,create a DSL seed job,copy*.groovy into the worksp

28、ace.Then build the seed job.JobDSLWhat?CI team decided to introduce JobDSL plugin.It allows us to describe our Jenkins jobs as Groovy scripts and store them as source code in the repository of the project.Why?As it has been already stated,it was a huge effort to maintain and supervise the jobs.By on

29、ly using the web interface of Jenkins,its almost impossible to track the changes of the jobs(Job config history plugin can help you with some basic information,but its still not a solution).Pipeline jobs are used by many people in the PT,communication of the changes in the jobs cannot be done via ma

30、il(if only mail was always sent out about the changes).Not to mention repeating/common part of the jobs,etc.The introduction of JobDSL plugin and Groovy scripts will reduce maintenance costs and decrease time to deliver new CI features.How can I learn more about this plugin?Start here:https:/ is a G

31、etting started section.Also note the Pages section on the right side of the wiki page.It contains a lot of useful links to pages like:FAQ-https:/ reference-https:/ reference-https:/ Configure Block-https:/ playground(https:/job- also useful place in case you want to test your Groovy scripts.编译加速 EA:

32、One Pager Report Significant improvements have been observed Up to 12 times faster builds for full clean build(200Mins vs.17Mins)Total Investment:240,000 SEK per year 50 EA Agents 5 Agent Servers RIO Payback Period:0.4 months0500full clean buildmakeemakeEA ArchitectureBuild Cluster(10-100

33、 CPUs)ClusterManagerDeveloper MachinesProduction MachinesNetworkAgentAgentElectric File SystemNodeKernelUser-levelBuild with ElectricMakeinstead of GNU MakeGerrit/Git RepositoryelectriccommanderelectricacceleratorelectricinsightAutomateAccelerateAnalyzeBuild Acceleration:Build History File55Each blo

34、ck is a job running on an agent for some durationWhite space isthe enemyCluster is idle while blocking job runs(serialization or stall)Y-Axis is AgentsX-Axis is TimeEA Conclusion 6 Hours to 90 Minutes by build avoidance.Build out 3rdparty platform from nightly build.Reuse 3rdparty platform in a new

35、workspace.90 Minutes to 10 Minutes by build acceleration.Research on EA and deploy it in build environment.EA requires specific configuration for a given project.The tuning of EA is a continuous activity.Other considerations.Create workspace on local disk will reduce build time by 50%.Revise makefil

36、e to invoke local compiler first.编译加速-虚拟内存盘 RAMDISK原理和用途 通过软件将一部分内存（RAM）模拟为硬盘来使用的一种技术。实现及软件编辑 DOS系统：XMSDSK Windows系统：VSuite Ramdisk Linux系统：直接格式化并挂载/dev/ramX 即可效果：提速一倍ccache Compiler Cache Wrapper for gcc Works fine for C,C+,Objective C or Objective C+Hash on source-code+header Local or Shared cache

37、 Can be combined with distcc Toolchain/crosscompiler may not support ccache.Precompiled headers Sometimes a single cpp-file needs hundreds of headerfiles Precompiled header flatten out headerfiles that includes many other headerfiles You can only have definitions in headerfiles,no implementationJenk

38、ins Builds Promotion and LockBuild Keeper PluginBuild Promotion PluginTo Keep a build for ever添加内容Code Change Auto-Review in GerritRun Gerrit Sanity Test on All Code Changes如果任何一项检查没有通过把你的代码存入Github通过gerrithub来reviewhttps:/review.gerrithub.io/static/intro.htmlSetup Gerrit in JenkinsJenkins SystemJen

39、kins JobRain MINGW64/src/iptnms-p/.git/hooks(GIT_DIR!)applypatch-msg.sample*pre-applypatch.sample*pre-push.sample*commit-msg*pre-commit.sample*pre-rebase.sample*commit-msg.sample*prepare-commit-msg*update.sample*post-update.sample*prepare-commit-msg.sample*关联代码修改与User Story和Bug Fix关联代码修改与User Story和

40、Bug Fixgit commit$cat.gitmodulessubmodule fuel/librarypath=fuel/libraryurl=././ecs/mirantis/openstack/fuel-librarybranch=cee/ecs-mossubmodule fuel/ostfpath=fuel/ostfurl=././ecs/mirantis/openstack/fuel-ostfbranch=cee/ecs-mosGit Submodules Auto UpliftJenkins Slave auto-manager Auto add a new slave lab

41、el for a new branch pipeline Keep the next build on the same slave Remove slave label when branch obsoletes Delete its workspace on slave after 20 daysArtifactory+Docker:包文件管理与发布Artifactory will empower your experience,especially with the following integrations.Working with Docker:http:/ with Git:ht

42、tp:/ with Jenkins:http:/wiki.jfrog.org/confluence/display/RTF/Jenkins+%28Hudson%29+Artifactory+Plug-inAnt+Ivy、Maven和Gradle这些构建工具都可以自动下载Artifactory里的构件（artifact），此外Jenkins、Bamboo等CI工具也可以通过构建工具将生成的构件（artifact）部署到Artifactory上Or Nexus配置ArtifactoryManage Jenkins-Configure System-ArtifactoryJob-Configure-

43、Generic-Artifactory IntegrationThen in Artifactory,the artifacts published.Uploading using Jenkins-ArtifactoryAll deliveries from components to Release Management shall be done by uploading the binary packages into Artifactory.This includes installation packages,installation scripts,change logs and/

44、or release notes,etc.Any file type is supported(zip,gz,tgz,etc.)Uploading using CLI-ArtifactoryGet ARM-Uploader and use it in your CLI.To get usage description,run:java-jar artifactoryUploader.jar-hUsage example:Suppose you want to upload a file named package.tar as version 1.0.0 of CEE delivery to

45、ARM.You would execute:java-jar artifactoryUploader.jar-repo proj-ecs-dev-local-p cee-r 1.0.0-usr -pwd -file/path/to/my/package/package.tarNOTE!Be sure to use your encrypted password(#Artifactory|see this section)Uploading using Web GUI-Artifactory1.Log in to ARM with your account2.Click Deploy and choose the file that you want to deploy 3.Enter the details as show in the picture below.4.Refer to Naming your binaries for more details on file naming采用容器和虚拟机使用VirtualBox代替OpenStackQ&A