1、#CiscoLive#CiscoLiveMax Ardica,Distinguished EngineerRamses Smeyers,Distinguished Engineer BRKDCN-2919How to Setup anACI Multi-Site withSingle Pod and Multi-Pod 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWho Are the Presenters?Max ArdicaDistinguished Engineer DC Teamm

2、axardicaRamses SmeyersDistinguished Engineer CX TACrsmeyers3BRKDCN-2919 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveEnter your personal notes hereCisco Webex App Questions?Use Cisco Webex App to chat with the speaker after the sessionFind this session in the Cisco Live

3、 Mobile AppClick“Join the Discussion”Install the Webex App or go directly to the Webex spaceEnter messages/questions in the Webex spaceHowWebex spaces will be moderated by the speaker until June 9,2023.12344https:/ 2023 Cisco and/or its affiliates.All rights reserved.Cisco PublicBRKDCN-29194Agenda 2

4、023 Cisco and/or its affiliates.All rights reserved.Cisco PublicWhat We Want to Achieve TodayPrerequisitesHardware Inspection and InstallationInstalling the First DC Site Expanding the Single Pod into a Multi-Pod FabricIntroducing Multi-Site and NDOAdding the DR Site on NDONDO Additional Functionali

5、tiesBRKDCN-29195 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveSite1Site1-Pod 1Pod 1Site 1Site 1What We Want to Achieve TodayWANStarting PointRamMax Inc.starts its business operations with a single DC locationClients can successfully access the application inside the DCB

6、RKDCN-29196 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveIPNSite1Site1-Pod 1Pod 1Site1Site1-Pod Pod 2 2Multi-Pod Fabric(Active/Active DCs)Site 1Site 1What We Want to Achieve TodayWANRamMaxs business is very successful and there is a need to grow the size of the DC fabri

7、cMulti-Pod is the architecture of choice to extend the ACI fabric presence across two DC locations in the same metro area(operated as active/active DCs)BRKDCN-29197 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveIPN/ISNSingle external network used for IPN and ISNNexus Das

8、hboard OrchestratorSite1Site1-Pod 1Pod 1Site1Site1-Pod Pod 2 2Multi-Pod Fabric(Active/Active DCs)Site 1Site 1What We Want to Achieve TodaySite2Site2-Pod 1Pod 1DR SiteWANSite 2Site 2RamMax decides to build a DR site to ensure business continuance and recoveryMulti-Site is the architecture used to int

9、erconnect the Prod DCs to the DR SiteBRKDCN-29198 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveIPN/ISNSingle external network used for IPN and ISNNexus Dashboard OrchestratorSite1Site1-Pod 1Pod 1Site1Site1-Pod Pod 2 2Multi-Pod Fabric(Active/Active DCs)Site 1Site 1What W

10、e Want to Achieve TodaySite2Site2-Pod 1Pod 1DR SiteWANSite 2Site 2RamMax decides to build a DR site to ensure business continuance and recoveryMulti-Site is the architecture used to interconnect the Prod DCs to the DR SiteX XX XBRKDCN-29199Prerequisites 2023 Cisco and/or its affiliates.All rights re

11、served.Cisco Public#CiscoLiveBefore starting,you should have:For each APIC a routable IP addresses for OOB mgmt and CIMC Functional NTP server Serial number of all leaf and spine nodes Optionally but recommended:1 IP per leaf and spine for OOB SCP/FTP/HTTP server(software)Console/serial server Infra

12、structure VLAN/VTEP pool vCenter IP address and credentialsPrerequisitesBRKDCN-291911Hardware Inspection and Installation 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveBRKDCN-291913 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveOur Setup for

13、Today(Detailed View)Bru ACI Fabric POD 37&38BDSOL-ACI37-SPINE1BDSOL-ACI37-LEAF1BDSOL-ACI37-APIC1BDSOL-ACI38-SPINE1 bdsol-aci37-router1 bdsol-aci37-server1 BDSOL-ACI37-SPINE2BDSOL-ACI37-SPINE3BDSOL-ACI37-SPINE4BDSOL-ACI37-LEAF2BDSOL-ACI37-LEAF3BDSOL-ACI37-LEAF4BDSOL-ACI37-APIC2BDSOL-ACI37-APIC3 bdsol

14、-aci37-router2 bdsol-aci37-server2BDSOL-ACI38-SPINE2BDSOL-ACI38-LEAF1BDSOL-ACI38-LEAF2 bdsol-aci38-server1 bdsol-aci38-router1BDSOL-ACI38-APIC1BDSOL-ACI38-APIC2BDSOL-ACI38-APIC3 bdsol-aci38-multisite1 bdsol-aci38-router2 bdsol-aci37-multipod1 multipod2 bdsol-aci37-multipod3multipod4 bdsol-aci38-serv

15、er2 bdsol-aci38-router3BRKDCN-291914Installing the First DC SiteSite1-Pod1 Configuration 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInstalling the First DC SiteSite1-Pod1:Initial Fabric Setup(Already Done)APIC initial configuration(S1P1-APIC1)only the 1stone for now1s

16、tleaf discoverySpines discovery2ndleaf discoveryS1P1-APIC2 configurationVerificationOOB mgmt IPs for leaf and spine nodesBRKDCN-291916 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInstalling the First DC SiteSite1-Pod1 FabricSite1-Pod 1S1P1-Spine201vCenter Server 1S1P1-

17、Spine202S1P1-Leaf101S1P1-Leaf102S1P1-APIC1S1P1-APIC2WAN192.168.200.100/30.101.102192.168.1.1ESXi Cluster192.168.100.10BRKDCN-291917 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveS1P1S1P1-APIC1APIC1S1P1S1P1-APIC2APIC2Fabric nameFabric1Fabric1Fabric ID11Active controllers3

18、3Pod ID11Controller ID12TEP Pool10.0.0.0/1610.0.0.0/16Infra VLAN39373937Installing the First DC SiteParameters for the APIC Initial Setup ScriptBRKDCN-291918 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveInstalling the First DC SiteSite1-Pod1:Remaining Configuration Step

19、s to DoNTP configuration Route Reflector for intra-BGP VPNv4 sessionsVMM integrationTenant configuration with Ecommerce running applicationEcommerce app connectivity verificationL3Out creation and external connectivity verificationBRKDCN-291919Demo 1NTP,VMM,L3Out Configuration and Pod VerificationEx

20、panding the Single Pod into a Multi-Pod Fabric 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveExpanding the Single Pod into a Multi-Pod FabricAdding the IPN and Site1-Pod2Step 1:verify the Inter-Pod Network(IPN)configurationStep 2:create the Multi-Pod fabric using the API

21、C WizardAdd Site1-Pod1Add Site1-Pod2Discovery of Pod2s leaf and spines nodesStep 3:S1P2-APIC3 in Pod2 joins the APIC clusterStep 4:extend Ecommerce Tenant to Pod2(L3Out,ESXi host,access policies)Verification Steps:Verify that the existing tenant configuration is extended into the Multi-Pod fabricVer

22、ify East-West and North-South connectivityBRKDCN-291922 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveExpanding the Single Pod into a Multo-Pod FabricAdding the IPN and Site1-Pod2Site1-Pod 1BDSOL-ACI37-APIC1BDSOL-ACI37-APIC2vCenter Server 1S1P1-Spine201S1P1-Spine202S1P1-

23、Leaf101S1P1-Leaf102Site1-Pod2BDSOL-ACI37-APIC3S1P2-Spine401 S1P2-Spine402S1P2-Leaf301S1P2-Leaf302IPNBDSOL-ACI37-Multipod3BDSOL-ACI37-Multipod4BDSOL-ACI37-Multipod1BDSOL-ACI37-Multipod2BRKDCN-291923 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveTEP Pool:TEP Pool:10.0.0.0/

24、1610.0.0.0/16Expanding the Single Pod into a Multi-Pod FabricStep1:Verify the Inter-Pod Network(IPN)ConfigurationS1P1-Spine201TEP Pool:TEP Pool:10.1.0.0/1610.1.0.0/16IPN1 Primary RP1/331/331/331/33.101.101.102.1021/341/341/341/34.109.109.110.110Site1-Pod1Site1-Pod2.1.1.5.5.9.9.13.13.2.2.6.6.14.14.10

25、.10IPN Infra Address Space:172.16.101.0/24.17.17.21.21.25.25.29.29.18.18.26.26.22.22.30.30IPN3,Backup RPIPN2IPN4S1P1-Spine202S1P2-Spine401S1P2-Spine402OSPF Area 0PIMPIMPIMPIM1/351/351/351/351/361/361/361/361/481/48PIMPIMPIMPIM1/481/481/481/481/481/48Pod1 External TEPPod1 External TEP-PoolPool172.16.

26、1.0/24172.16.1.0/24Pod2 External TEPPod2 External TEP-PoolPool172.16.2.0/24172.16.2.0/24BRKDCN-291924Demo 2Verify the Inter-Pod Network(IPN)ConfigurationDemo 3Create the Multi-Pod Fabric Using the APIC Wizard 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveExpanding the Si

27、ngle Pod into a Multi-Pod FabricStep2:Create the Multi-Pod Fabric Using the APIC Wizard and import Pod2 Spine and Leaf NodesNode IDNode IDPod IDPod IDNameNameS/NS/N3012S1P2-Leaf301FDO224702ET3022S1P2-Leaf302FDO223007J44012S1P2-Spine401FDO22472FCV4022S1P2-Spine402FDO22391NP2Nodes automatically discov

28、ered in Site1-Pod2 that need to be added to the APIC fabric membership tableBRKDCN-291927 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveS1P1S1P1-APIC1APIC1S1P1S1P1-APIC2APIC2S1P2S1P2-APIC3APIC3Fabric nameFabric1Fabric1Fabric1Fabric ID111Active controllers333Pod ID112Cont

29、roller ID123TEP Pool10.0.0.0/1610.0.0.0/1610.0.0.0/16Infra VLAN393739373937Pod2 uses TEP Pool 10.1.0.0/16 but S1P2-APIC3 resides in TEP Pool of Pod1Expanding the Single Pod into a Multi-Pod FabricStep 3:S1P2-APIC3 in Pod2 Joins the APIC ClusterBRKDCN-291928 2023 Cisco and/or its affiliates.All right

30、s reserved.Cisco Public#CiscoLiveTEP Pool:TEP Pool:10.0.0.0/1610.0.0.0/16Expanding the Single Pod into a Multi-Pod FabricFull OSPF Adjacencies between Spines and IPN in both PodsS1P1-Spine201TEP Pool:TEP Pool:10.1.0.0/1610.1.0.0/16IPN1 Primary RP1/331/331/331/33.101.101.102.1021/341/341/341/34.109.1

31、09.110.110Site1-Pod1Site1-Pod2.1.1.5.5.9.9.13.13.2.2.6.6.14.14.10.10IPN Infra Address Space:172.16.101.0/24.17.17.21.21.25.25.29.29.18.18.26.26.22.22.30.30IPN3,Backup RPIPN2IPN4S1P1-Spine202S1P2-Spine401S1P2-Spine402OSPF Area 0PIMPIMPIMPIM1/351/351/351/351/361/361/361/361/481/48PIMPIMPIMPIM1/481/481

32、/481/481/481/48Pod1 External TEPPod1 External TEP-PoolPool172.16.1.0/24172.16.1.0/24Pod2 External TEPPod2 External TEP-PoolPool172.16.2.0/24172.16.2.0/24BRKDCN-291929Demo 4S1P2-APIC3 in Pod2 Joins the APIC Cluster 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveExpanding t

33、he Single Pod into a Multi-Pod FabricStep 4:Extend Ecommerce/Tenant to Pod2(L3Out,ESXi Host to VDS,etc.)Site1-Pod1WANSite1-Pod21/171/171/171/171/191/191/191/19.1.1.2.2.5.5.6.6.9.9.10.10.13.13.14.141/171/171/171/171/191/191/191/191/471/471/71/71/471/471/91/9.101.101.105.105.106.106.102.102Ecommerce T

34、enant External Address Space192.168.200.0/241/11/1Site1-L3OutConnectivity to the WAN network is pre-provisionedStretched ESXi Cluster1/111/111/111/111/111/111/111/11192.168.100.10/24BRKDCN-291931Demo 5Extend Ecommerce tenant configuration to Pod2Introducing NDO and Multi-Site 2023 Cisco and/or its a

35、ffiliates.All rights reserved.Cisco Public#CiscoLiveIntroducing NDO and Multi-SiteConfiguration StepsInitial setup of ND/NDOAdding the Multi-Pod fabric as first site on ND/NDOOnboarding the Multi-Pod Fabric on NDSetting the Multi-Pod Fabric as“Managed”on NDOImporting existing Ecommerce tenant config

36、uration on NDOBRKDCN-291934 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveSite1Site1-Pod 1Pod 1IPN/ISNWANSite1Site1-Pod Pod 2 2O O-MTEPMTEP-S1S1O O-UTEPUTEP-S1P1S1P1O O-UTEPUTEP-S1P2S1P2O-UTEP-S1P1:172.16.100.101O-UTEP-S1P2:172.16.100.102O-MTEP-S1:172.16.100.100BGP Speak

37、er 1:172.16.100.201BGP Speaker 2:172.16.100.202BGP Speaker 1BGP Speaker 1BGP Speaker 2BGP Speaker 2Site1Site1-L3OutL3OutAdding the Multi-Pod Fabric on NDOAssign Multi-Site TEP Addresses and BGP EVPN Router-IDsSite 1Site 1BRKDCN-291936 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public

38、#CiscoLiveSite 1Schema Design(Initial State)One Template for Site 1 local objects,plus Stretched TemplatesEPG1EPG2BD1BD2C1Site 1 Template(Ecommerce)Contract-VRF-Stretched(Ecommerce)Schema Ecommerce Schema Ecommerce ANP1L3Out-S1VRFEPG-EEPG-BD-Stretched(Ecommerce)Ext-EPGBRKDCN-291937Demo 6Simplified T

39、enant Management through NDOAdding the DR Site on NDO 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveThrough automation,create the necessary configuration for the DR siteAccess policies VMM domain integrationAdding the DR Site on NDODR Site PreparationBRKDCN-291940 2023 C

40、isco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAdding the DR Site on NDOConfiguration StepsAdding the DR fabric as a second site on ND/NDO(onboarding it on ND,managing it on NDO,assigning Multi-Site TEP addresses,BGP EVPN Router-IDs and External TEP pool,establishing OPSF conne

41、ctivity between spines and IPN routers)Verifying IPN connectivityExtending the tenant Ecommerce to the DR site Create the Site2 local L3Out directly on NDO in the“L3OUt Template”(new NDO 4.1(1)feature)Extending the existing Ecommerce tenant configuration tenant configuration to the DR siteVerify Eas

42、t-West and North-South connectivityBRKDCN-291941 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAdding the DR Site on NDOSite2-Pod1 FabricSite2Site2-Pod1Pod1S2P1-Spine201BDSOL-ACI38-APIC1BDSOL-ACI38-APIC2BDSOL-ACI38-APIC3S2P1-Spine202S2P1-Leaf101S2P1-Leaf102ESXi Cluster 2

43、BRKDCN-291942 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveS2P1S2P1-APIC1APIC1S2P1S2P1-APIC2APIC2S2P1S2P1-APIC3APIC3Fabric nameFabric2Fabric2Fabric2Fabric ID111Active controllers333Pod ID111Controller ID123TEP Pool10.2.0.0/1610.2.0.0/1610.2.0.0/16Infra VLAN393739373937A

44、dding the DR Site on NDOParameters for the APIC Initial Setup Script(Already Done)Recommended not to use overlapping TEP Pools with existing sitesBRKDCN-291943 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveBGP AS 65001Site1Site1-Pod 1Pod 1IPN/ISNSite 1Site 1Site 2Site 2A

45、dding the DR Site on NDOAssign Routable TEP Addresses and BGP EVPN Router-IDSWANSite1Site1-Pod Pod 2 2Site2Site2-Pod 1Pod 1O O-MTEPMTEP-S1S1O O-UTEPUTEP-S1P1S1P1O O-UTEPUTEP-S1P2S1P2O O-MTEPMTEP-S2S2O O-UTEPUTEP-S1P1S1P1O-UTEP-S1P1:172.16.200.101O-MTEP-S1:172.16.200.100BGP Speaker 1:172.16.200.201BG

46、P Speaker 2:172.16.200.202External TEP Pool:172.16.3.0/24BGP Speaker 1BGP Speaker 1BGP Speaker 2BGP Speaker 2BGP Speaker 1BGP Speaker 1BGP Speaker 2BGP Speaker 2Site1Site1-L3OutL3OutSite2Site2-L3OutL3OutNexus Dashboard OrchestratorBGP AS 65002O-UTEP-S1P1:172.16.100.101O-UTEP-S1P2:172.16.100.102O-MTE

47、P-S1:172.16.100.100BGP Speaker 1:172.16.100.201BGP Speaker 2:172.16.100.202BRKDCN-291944 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAdding the DR Site on NDOVerifying IPN connectivityIPN11/481/481/481/48.1.11/481/481/481/48.5.5IPN3IPN2IPN41/11/11/21/21/31/31/41/4WAN I

48、nfra Address Space:172.16.110.0/24.2.2.6.6.9.9.10.10.13.13.14.141/51/5.18.18TEP Pool:TEP Pool:10.2.0.0/1610.2.0.0/16Site2-Pod1.2.2.6.6.1.1.5.5IPN Site2 Infra Address Space:172.16.102.0/241/51/5.17.17IPN5S2P1-Leaf201S2P1-Leaf202BRKDCN-291945Demo 7Adding the DR Site on NDO 2023 Cisco and/or its affili

49、ates.All rights reserved.Cisco Public#CiscoLiveAdding the DR Site on NDOCreate a Local L3Out in the DR Site Directly on NDOSite2Site2-Pod1Pod1.17.17.18.18.21.21.22.221/171/171/171/171/191/191/191/191/471/471/111/11.109.109.110.110Ecommerce Tenant External Address Space192.168.200.0/24(VLAN Tag 901)1

50、/11/1192.168.100.10/24Site2Site2-L3OutL3OutS2P1-Leaf101(RID:192.168.200.205)S2P1-Leaf102(RID:192.168.200.206)BRKDCN-291947 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveCreate a Local L3Out Directly on NDOCross-Referencing Different Types of TemplatesL3Out TemplateInterf

51、ace routing policyInterface routing policyOSPF Interface PolicyTenant Policy TemplateL3Out1L3Out1App TemplateVRF1VRF1L3Out1L3Out1VRFVRFProtocol:Protocol:OSPFInterface Group Interface Group Policy Pol1Policy Pol1NodesNodesPolicyPolicy101Pol1102Pol2InterfacesInterfacesPolicyPolicy101/1/1Pol1102/1/1Pol

52、1BD1BD1BRKDCN-291948 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveSite 1Schema Design(Today)One Template for Site 1 local objects,plus Stretched TemplatesEPG1EPG2BD1BD2C1ANP1L3Out-S1VRFExt-EPGBRKDCN-291949Schema Ecommerce Schema Ecommerce Site 1 Template(Ecommerce)Contr

53、act-VRF-Stretched(Ecommerce)EPG-EEPG-BD-Stretched(Ecommerce)2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveSite 1Schema Design(Final State)Migrating EPGs/BDs to the Stretched TemplateEPG1EPG2BD1BD2C1ANP1L3Out-S1VRFSite 2Ext-EPGBRKDCN-291950Schema Ecommerce Schema Ecommerc

54、e Site 1 Template(Ecommerce)Contract-VRF-Stretched(Ecommerce)EPG-EEPG-BD-Stretched(Ecommerce)NDO Additional Functionalities 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveNDO Additional FunctionalitiesEnd host connectivity verificationHost route advertisement(inbound traf

55、fic optimization)BRKDCN-291952Demo 8NDO Additional Functionalities 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveACI Multi-Pod White Paperhttp:/ Multi-Pod Configuration Paperhttps:/ Multi-Pod and Service Node Integration White Paperhttps:/ Multi-Site White Paperhttps:/ M

56、ulti-Site Deployment Guide for ACI Fabricshttps:/ Multi-Site and Service Node Integration White Paperhttps:/ to Go for More InformationBRKDCN-291954 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveFill out your session surveys!Attendees who fill out a minimum of four sessi

57、on surveys and the overall event survey will get Cisco Live-branded socks(while supplies last)!These points help you get on the leaderboard and increase your chances of winning daily and grand prizesAttendees will also earn 100 points in the Cisco Live Challenge for every survey completed.BRKDCN-291

58、955 2023 Cisco and/or its affiliates.All rights reserved.Cisco PublicContinue your educationVisit the Cisco Showcase for related demosBook your one-on-oneMeet the Engineer meetingAttend the interactive education with DevNet,Capture the Flag,and Walk-in LabsVisit the On-Demand Library for more sessio

59、ns at www.CiscoL you#CiscoLive 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLive58Gamify your Cisco Live experience!Get points Get points for attending this session!for attending this session!Open the Cisco Events App.Click on Cisco Live Challenge in the side menu.Click on View Your Badges at the top.Click the+at the bottom of the screen and scan the QR code:How:123458 2023 Cisco and/or its affiliates.All rights reserved.Cisco PublicBRKDCN-2919#CiscoLive