《思科 ACI 多容器设计和部署.pdf》由会员分享,可在线阅读,更多相关《思科 ACI 多容器设计和部署.pdf(80页珍藏版)》请在三个皮匠报告上搜索。
1、#CiscoLive#CiscoLiveJohn Weston,Technical Marketing Engineer,Data Center NetworkingBRKDCN-2949Design and DeploymentCisco ACI Multi-Pod 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveSession Objectives At the end of the session,the participants should be able to:Articulate
2、 the different deployment options to interconnect Cisco ACI networks(Multi-Pod and Multi-Site)and when to choose one vs.the otherUnderstand the functionalities and specific design considerations associated to the ACI Multi-Pod architectureInitial assumption:The audience already has a good knowledge
3、of ACI main concepts(Tenant,BD,EPG,L2Out,L3Out,etc.)3BRKDCN-2949 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveEnter your personal notes hereCisco Webex App 4Questions?Use Cisco Webex App to chat with the speaker after the sessionFind this session in the Cisco Live Mobil
4、e AppClick“Join the Discussion”Install the Webex App or go directly to the Webex spaceEnter messages/questions in the Webex spaceHowWebex spaces will be moderated by the speaker until June 9,2023.12344https:/ 2023 Cisco and/or its affiliates.All rights reserved.Cisco PublicBRKDCN-2949#CiscoLive 2023
5、 Cisco and/or its affiliates.All rights reserved.Cisco PublicAgendaOverview,Use Cases,and supported TopologiesAPIC Cluster DeploymentInter-Pod ConnectivityControl and Data PlanesConnecting to External NetworksNetwork Services IntegrationRemote LeafBRKDCN-29495Overall Design Principles(AZs and Region
6、s)2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveACI Single Pod Fabric ACI 1.0-Leaf/Spine Single Pod FabricACI 2.0-Multiple Networks(Pods or Availability Zones)in a single Fabric(Region)Pod AMP-BGP-EVPNIPNPod nACI Multi-Pod FabricAPIC ClusterACI 3.1/4.0-Remote Leaf extend
7、s a Fabric to remote locationsACI Remote LeafACI Fabric and Policy Domain Evolution7ACI 3.0 Multiple Fabrics(Regions)interconnected in the same Multi-Site Orchestrator domainFabric AMP-BGP-EVPNISNFabric nACI Multi-SiteACI 4.1&4.2 ACI Extensions to Public Cloud Cloud Network ControllerBRKDCN-2949 202
8、3 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveSystems View(How do these things relate)Change and Network Fault Domain IsolationPod B.1(AZ 1)Pod B.2(AZ 2)Multi-Pod Fabric B(Region 2)Pod A.1(AZ 1)Pod A.2(AZ 2)Multi-Pod Fabric A(Region 1)Application Policy Change DomainCommon
9、Namespace(IP,DNS,Active Directory)Fabric Network Fault DomainActive WorkloadsLayer 2&Layer 3Layer 3Inter RegionApplication Policy Change DomainActive WorkloadsLayer 2&Layer 3Fabric Network Fault DomainFabric Network Fault DomainFabric Network Fault DomainApplication1 workloads deployed across Pods(A
10、Zs)Application 2 workloads deployed across Pods(AZs)8BRKDCN-2949Nexus Dashboard Orchestrator 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveData Center 2Pod 1.A AZ1Pod 2.A AZ2Classic Active/Active(L2 and L3)Multi-Pod Fabric A(Region 1)Pod 1.B AZ1Pod 2.B AZ2Multi-Pod Fabri
11、c B(Region 2)Classic Active/Active(L2 and L3)Data Center 1Nexus Dashboard OrchestratorL3 OnlyL3 OnlyMulti-Pod+Multi-Site Satisfying Conflicting Requirements(A/A DCs and DR)BRKDCN-29499 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLive 50 msec RTTAPIC ClusterForwarding contr
12、ol plane(IS-IS,COOP)fault isolationData Plane VXLAN encapsulation between PodsEnd-to-end policy enforcementPod APod APod nPod nACI Multi-PodThe Ideal Architecture for Active/Active DC DeploymentsVXLANIS-IS,COOP,MP-BGPIS-IS,COOP,MP-BGPMultiple ACI Pods connected by an IP Inter-Pod L3 network,each Pod
13、 consists of leaf and spine nodesManaged by a single APIC ClusterSingle Management and Policy DomainMPMP-BGP BGP-EVPNEVPNInterInter-Pod NetworkPod NetworkBRKDCN-294910Availability Zone AAvailability Zone BRegion 1ACI Multi-Pod Deep Dive Overview,Use Casesand Supported Topologies 2023 Cisco and/or it
14、s affiliates.All rights reserved.Cisco Public#CiscoLive13Multi-Pod Supported TopologiesBRKDCN-2949IPNPods connected via an InterPods connected via an Inter-Pod Network(IPN)Pod Network(IPN)Pods directly connected without an IPN(from 5.2(3)and later)Pods directly connected without an IPN(from 5.2(3)an
15、d later)(up to 50 mss RTT)(up to 50 mss RTT)(up to 50 mss RTT)(up to 50 mss RTT)Pod1Pod1Pod2Pod2Pod1Pod1Pod2Pod2 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveMulti-Pod Spines Back-to-Back 14BRKDCN-2949Many customers deploy ACI Multi-Pod fabrics with only two Pods and ar
16、e not using any other features that require spine IPN connectivity(Multi-Site,Remote leaf,GOLF,Cloud ACI)These customers may have small to medium size fabrics and the requirement to build an additional network(IPN)for inter-Pod connectivity is an added cost(plus the need to operate it)The ACI Multi-
17、Pod Spines back-to-back option removes the requirement to build and operate an additional network for inter-Pod connectivityACI Release 5.2(3)2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveMulti-Pod Spines Back-to-Back Guidelines and Restrictions15BRKDCN-2949Support is li
18、mited to a topology with 2 Pods leveraging 2ndgeneration spines onlyOSPF underlay peering,MP-BGP overlay peering between the spines in separate PodsNo need for PIM-Bidir(spines do not run PIM)MACsec encryption supported across PodsNot compatible functionsACI Multi-SiteRemote LeafGOLFCloud ACIAPIC co
19、nnectivity via L3 networkBack-to-Back+IPN only supported for migration purposes(migration is disruptive)IPNBack-to-Back+IPN(Migration Only)2ndGen SpinesOSPF+MP-BGPMACsec 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveMulti-Pod Spines Back-to-Back Supported Topologies16BRK
20、DCN-2949LLDPBack-to-back spine connectivity must be point-to-point(physical or logical)Spines discover back-to-back connections via LLDPLinks can be directly connected or must support tunneling of LLDP packets PWLLDP over L2TP 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLi
21、veMulti-Pod Spines Back-to-Back Supported Topologies17BRKDCN-2949Full mesh between spinesRecommendedIt is not mandatory for all spines in a Pod to connect to all the spines in the other Pod,the design decision must be made based on resiliency/bandwidth considerationsPartial mesh between spinesSuppor
22、ted 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAll existing Nexus 9000 HW supported as leaf and spine nodes*Maximum number of supported ACI leaf nodes(across all Pods)Up to 80 leaf nodes supported with a 3 node 3 node APIC cluster200 leaf nodes(across Pods)with a 4 no
23、de 4 node APIC cluster(from ACI release 4.1)300 leaf nodes(across Pods)with a 5 node 5 node APIC Cluster400 leaf nodes(across Pods)with a 7 node 7 node APIC Cluster(from ACI release 2.2(2e)500 leaf nodes(across Pods)with a 7 node 7 node APIC Cluster(from ACI release 4.2(4)Maximum 400 leaf nodes per
24、Pod(from ACI release 4.2(4)Up to 6 spines per Pod,50 spines per Fabric(from ACI release 6.0(1)Maximum number of supported Pods 4 in 2.0(1)/2.0(2)releases 6 in 2.1(1)release 10 in 2.2(2e)release12 in 3.0(1)release25 in 6.0(1)releaseACI Multi-Pod18BRKDCN-2949SW/HW Support and Scalability ValuesAPIC Cl
25、uster Deployment Considerations 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveThe Data Base is replicated across APIC nodesShard 2Shard 2Shard 1Shard 1Shard 3Shard 3Shard 1Shard 1Shard 1Shard 1Shard 2Shard 2Shard 2Shard 2Shard 3Shard 3Shard 3Shard 3One copy is active for
26、 every specific portion of the Data BaseProcesses are active on all nodes(not active/standby)The Data Base is distributed as active+2 backup instances(shards)for every attributeAPIC Distributed Multi-Active Data BaseBRKDCN-294920 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#Cisc
27、oLiveAPIC will allow read-only access to the DB when only one node remains active(standard DB quorum)Hard failure of two nodes cause all shards to be in read-only mode(of course reboot etc.heals the cluster after APIC nodes are up)Additional APIC will increase the system scale(up to 7*nodes supporte
28、d)but does not add more redundancyHard failure of two nodes would cause inconsistent behaviour across shards(some will be in read-only mode,some in read-write mode)Shards in read-only modeShards in read-only modeShards in read-write modeAPIC Cluster Deployment Considerations21BRKDCN-2949Single Pod S
29、cenarioXXXX 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLivePod 1Pod 2Pod 1Pod 2Up to 50 msecXPod isolation scenario:changes still possible on APIC nodes in Pod1 but not in Pod2Pod hard failure scenario:recommendation is to activate a standby node to make the cluster fully
30、 functional againPod isolation scenario:same considerations as with single Pod(different behaviour across shards)Pod hard failure scenario:may cause the loss of information for the shards replicated across APIC nodes in the failed PodPossible to restore the whole fabric state to the latest taken con
31、figuration snapshot(ID Recovery procedure needs BU and TAC involvement)XX XXXAPIC Cluster Deployment Considerations22BRKDCN-2949Multi-Pod 2 Pods ScenarioX XXRead/WriteRead OnlyUp to 50 msec 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLivePod 1Pod 2Up to 50 msecXX XXInterme
32、diate scalability values compared to a 3 or 5 nodes cluster scenario(up to 200 leaf nodes supported)Pod isolation scenario:same considerations as with 5 nodes(different behaviour across shards)Pod hard failure scenarioNo chance of total loss of information for any shardCan bring up a standby node in
33、 the second site to regain full majority for all the shardsAPIC Cluster Deployment Considerations What about a 4 Nodes APIC Cluster?23BRKDCN-2949ACI Release 4.1(1)2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveMain recommendation:deploy a 3 nodes APIC cluster when less th
34、an 80 leaf nodes are deployed across PodsFrom 4.1(1)can deploy 4 nodes if the scalability requirements are metWhen 5(or 7)nodes are really needed for scalability reasons,follow the rule of thumb of never placing more than two APIC nodes in the same Pod(when possible):Pod1Pod2Pod3Pod4Pod5Pod62 Pods*3
35、 Pods4 Pods5 Pods6+Pods*ID Recovery procedure possible for recovering of lost informationAPIC Cluster Deployment Considerations24BRKDCN-2949Deployment RecommendationsAPIC Connectivity over L3 Network 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC Connectivity Options
36、APIC Cluster directly connected to fabricBRKDCN-2949Pod 1TEP Pool10.1.0.0/16Pod 2TEP Pool10.2.0.0/1610.1.0.1 10.1.0210.1.0.3IPNAPICs can be placed in any podAPIC fabric IP addresses are always assigned from pod 1 TEP poolRecommended to distribute APICs across pods so loss of a pod does not bring dow
37、n the entire cluster26 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC Connectivity OptionsAPIC cluster connected over L3 NetworkBRKDCN-2949Pod 1TEP Pool10.1.0.0/16Pod 2TEP Pool10.2.0.0/16IPNAPICs do not need to be directly connected to the leaf switches.Can be placed
38、 in L3 network that has IP reachability to the spines via IPNAPICs will be part of pod 0.Pod 0 is a special pod that only contains APICs and no fabric switchesAPIC fabric IP addresses are user configurable.Not assigned from any pod TEP rangeAPIC fabric IPs can be in the same or different subnet per
39、APICAPICs can be geographically distributed within the Multi-Pod 50 msec distance requirementPod 010.50.1.110.50.2.110.50.3.1ACI Release 5.2(1)27 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC Connectivity OptionsAPIC cluster connected over L3 Network,Secure Zone Use
40、 CaseBRKDCN-2949Pod 1TEP Pool10.1.0.0/16Pod 2TEP Pool10.2.0.0/16IPNAPIC Cluster over L3 Network supports use case where all traffic between APIC and switches must be inspected by a firewallAPIC cluster can be placed in a secure zone where all traffic into and out of the zone is inspected by a firewa
41、llPod 010.50.0.110.50.2.110.50.3.1ACI Release 5.2(1)APIC to fabric traffic can be inspected by firewall28 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC Connectivity OptionsAPIC cluster connected over L3 Network,IPN Multicast RequirementBRKDCN-2949Pod 1TEP Pool10.1.0
42、.0/16Pod 2TEP Pool10.2.0.0/16Multicast(PIM Bidir)is only required for inter-pod BUM trafficIf APIC cluster over L3 network is managing only one pod,multicast is not required in the IPNIf it is a Multi-Pod fabric,multicast is only required on the links interconnecting the podsPod 010.50.0.110.50.2.1
43、10.50.3.1ACI Release 5.2(1)PIMPIMPIM29 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC Connectivity OptionsVirtual APIC clusterBRKDCN-2949Pod 1TEP Pool10.1.0.0/16Pod 2TEP Pool10.2.0.0/16IPNVirtual APIC cluster(all virtual APICs)Runs as a VM on an ESXi hypervisorESXi s
44、erver directly connected to fabricNo mixed cluster support.Must be all virtual or all physicalSupports all types of deployments,Remote Leaf,Multi-Pod,Multi-Site.ACI Release 6.0(2)ESXi ServerESXi ServerESXi Server30 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveTopology c
45、onsiderations for virtual APIC on ESXiDirectly AttachedESXi servers need to be connected directly to ACI leaf nodes via individual links or vPC.(APIC1 must use Active-Standby instead of Active-Active with vPC)LLDP must be disabled on the virtual switch for LLDP discovery between leaf nodes and vAPIC
46、s.vmnic0vmnic1Leaf nodesDistributed Virtual SwitchSpine nodesvmnic0vmnic1vmnic0vmnic1Leaf nodesDistributed Virtual SwitchSpine nodesESXi serversvmnic0vmnic1vPCLLDP discoveryLLDP discoveryESXi serversLLDP discoveryLLDP discoveryBRKDCN-294931 2023 Cisco and/or its affiliates.All rights reserved.Cisco
47、Public#CiscoLiveDistributed Switch configurationPort Group VLAN configurationVLAN type:VLAN TrunkingVLAN trunk range:VLAN 0(VLAN 0 is required for APIC LLDP discovery)ACI Infra VLAN(for example,3914 is used as the default value during APIC initial setup)Inband VLAN(s)(VLAN 10 in the example)BRKDCN-2
48、94932 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC Connectivity OptionsVirtual APIC cluster over L3 NetworkBRKDCN-2949Pod 1TEP Pool10.1.0.0/16Pod 2TEP Pool10.2.0.0/16IPNVirtual APIC over L3 NetworkSame or different IP addresses per APIC same as physical APIC over L
49、3 networkCannot mix virtual APIC over L3 Network with directly connected virtual APICACI Release 6.0(2)ESXi Server ESXi Server ESXi ServerPod 033Inter-Pod Connectivity Deployment Considerations 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveNot managed Not managed by APIC
50、,must be separately configured(dayby APIC,must be separately configured(day-0 configuration)0 configuration)IPN topology can be arbitrary,not mandatory to connect to all spine nodesIPN topology can be arbitrary,not mandatory to connect to all spine nodesMain requirements:Main requirements:Multicast
51、BiDir PIM needed to handle Layer 2 BUM*trafficOSPF or BGP to peer with the spine nodes and learn VTEP reachability Increase MTU support to handle VXLAN encapsulated trafficDHCP-Relay*Broadcast,Unknown unicast,MulticastACI Multi-Pod Inter-Pod Network(IPN)Requirements35BRKDCN-2949 2023 Cisco and/or it
52、s affiliates.All rights reserved.Cisco Public#CiscoLiveBGP Underlay Support for IPN links36BRKDCN-294950 msec RTTAPIC ClusterMPMP-BGP BGP-EVPNEVPNACI Release 5.2(3)eBGPeBGPFrom ACI 5.2(3)you can use either OSPF and/or BGP for IPN connectivityInfra L3Out interfaces can be configured with OSPF,BGP,or
53、both protocols at the same time(typically used for migration)Only eBGP is supportedSupported for Multi-Pod,Remote Leaf,Multi-Site,and APIC over L3 NetworkWhen both protocols are configured,BGP routes will be preferred due to lower admin distance 2023 Cisco and/or its affiliates.All rights reserved.C
54、isco Public#CiscoLiveBGP Underlay Support for IPN links37BRKDCN-294950 msec RTTAPIC ClusterMPMP-BGP BGP-EVPNEVPNACI Release 5.2(3)eBGPeBGPfeature bgprouter bgp 65010router-id 10.10.10.1vrf IPNaddress-family ipv4 unicastneighbor 10.1.1.1 remote-as 65001address-family ipv4 unicastdisable-peer-as-check
55、Sample IPN configuration(Nexus 9000)10.1.1.1/3110.1.1.0/31Fabric BGP 65001IPN BGP 65010Configure BGP disable-peer-as-check if Nexus switches are used for IPNNexus switches will not advertise prefixes to peer if peer AS is already in the AS PATH.disable-peer-as-check turns off this behavior 2023 Cisc
56、o and/or its affiliates.All rights reserved.Cisco Public#CiscoLive1.Data Plane MTU:MTU of the traffic generate by endpoints(servers,routers,service nodes,etc.)connected to ACI leaf nodesNeed to account for 50B of overhead(VXLAN encapsulation)for inter-Pod communication2.Control Plane MTU:for CPU gen
57、erated traffic like EVPN across sitesThe default value is 9000B,9000B,can be tuned to the maximum MTU value supported in the ISNACI Multi-Pod and MTUDifferent MTU Meanings38BRKDCN-294950 msec RTTAPIC Cluster112MP-BGP EVPNIPN 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLive
58、Control Plane MTU can be set leveraging the“CP MTU Policy”on APICThe required MTU in the IPN would then depend on this setting and on the Data Plane MTU configurationAlways need to consider the VXLAN encapsulation overhead for data plane traffic(50/54 bytes)Modify the default 9000B MTU valueACI Mult
59、i-Pod and MTUTuning CP MTU for EVPN Traffic across Pods39BRKDCN-294950 msec RTTAPIC ClusterMPMP-BGP BGP-EVPNEVPN 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveTraffic across sites should be consistently prioritized(as it happens intra-site)To achieve this end-to-end cons
60、istent behavior,it is required to configure DSCP-to-CoS mapping in the infra Tenant Allows to classify traffic received on the spines from the IPN based on outer DSCP valueWithout the DSCP-to-CoS mapping configuration,classification for the same traffic will be CoS based(preserving CoS value in the
61、IPN is harder)The traffic can also then be properly treated inside the IPN(classification/queuing)Recommended to always prioritize at least Policy and Control Plane trafficTraffic classification and queuingACI Multi-Pod and QoSInter-Pod QoS Behavior40BRKDCN-2949APIC ClusterMPMP-BGP BGP-EVPNEVPNIPNCo
62、ntrol and Data Planes 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveDiscovery and provisioning of all the devices in the local Pod2Provisioning interfaces on the spines facing the IPN and EVPN control plane configuration3Spines in Pod 2 connect to the IPN and generate DH
63、CP requests4DHCP requests are relayed by the IPN devices back to the APIC in Pod 15DHCP response reaches Pod 2 spine allowing its full provisioning6Seed Pod 1Seed Pod 1Single APIC ClusterAPIC Node 2 joins the Cluster9Discovery and provisioning of all the devices in the local Pod7APIC Node 2 connecte
64、d to a Leaf node in Pod 28 1APIC Node 1 connected to a Leaf node in Seed Pod 1Pod 2Pod 210Discover other Pods following the same procedureACI Multi-PodAuto-Provisioning of Pods42BRKDCN-2949ip dhcp relay address 10.0.0.1ip dhcp relay address 10.0.0.2ip dhcp relay address 10.0.0.3 2023 Cisco and/or it
65、s affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC ClusterSeparate IP address pools for VTEPs assigned by APIC to each PodSummary routes advertised toward the IPN via OSPF or BGP routingIS-IS convergence events local to a Pod not propagated to remote PodsSpine nodes redistribute other Pods
66、summary routes into the local IS-IS process Needed for local VTEPs to communicate with remote VTEPsOSPF/BGPOSPF/BGP10.0.0.0/1610.1.0.0/16Leaf routing tableLeaf routing tableIP PrefixIP PrefixNextNext-HopHop10.1.0.0/16Pod1-S1,Pod1-S2ISIS-IS to IS to OSPF/BGP OSPF/BGP mutual mutual redistributionredis
67、tributionExchanging TEP information across pods43BRKDCN-294910.0.0.0/1610.1.0.0/16IPN Network Routing TableIPN 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC ClusterDefault fabric wide IS-IS metric is set at 63(max value)During upgrade,spines set the overload mode wh
68、ile policy is being downloadedIf fabric-wide value is already using the maxvalue for routes redistributed into IS-IS,the overload functionality is ineffectiveThis can create unexpected traffic interruption if leaf sends traffic to a spine which is not fully upgraded(and ready to forward traffic)OSPF
69、/BGPOSPF/BGP10.0.0.0/1610.1.0.0/16ISIS-IS to IS to OSPF/BGP OSPF/BGP mutual mutual redistributionredistributionExchanging TEP information across podsIssue with Default IS-IS Metric Policy Configuration 44BRKDCN-294910.0.0.0/1610.1.0.0/16IPN Network Routing TableIPNNode Upgrade Group 1(in hold down)R
70、edistribute remote TEP routes with IS-IS metric 64Leaf routing tableLeaf routing tableIP PrefixIP PrefixNextNext-HopHop10.1.0.0/16Pod1-S1,Pod1-S2Settings ISIS Policy(Default Config)2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC ClusterBy lowering the default ISIS met
71、ric value,connectivity to TEP prefixes received from the remote site will be preferred through the remaining spinesThis behavior gives time to the spine for completing the upgrade OSPF/BGPOSPF/BGP10.0.0.0/1610.1.0.0/16ISIS-IS to IS to OSPF/BGP OSPF/BGP mutual mutual redistributionredistributionExcha
72、nging TEP information across podsLowering the Default IS-IS Metric Policy45BRKDCN-294910.0.0.0/1610.1.0.0/16IPN Network Routing TableIPNNode Upgrade Group 1(in hold down)Redistribute remote TEP routes with IS-IS metric 64Leaf routing tableLeaf routing tableIP PrefixIP PrefixNextNext-HopHop10.1.0.0/1
73、6Pod1-S2Set this value to ISIS Policy 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveACI Multi-PodInter-Pod MP-BGP EVPN Control Plane46BRKDCN-2949MP-BGP EVPN to sync Endpoint(EP)and Multicast Group informationAll remote Pod entries associated to a Proxy VTEP next-hop addr
74、ess(not part of local TEP Pool)Same BGP AS across all the PodsiBGP EVPN sessions between spines in separate PodsFull mesh MP-iBGP EVPN sessions between local and remote spines(default behavior)Optional RR deployment(recommended one RR in each Pod for resiliency)MPMP-BGP BGP-EVPNEVPNEP1EP1EP1Leaf 1Le
75、af 1Proxy AProxy BEP1EP1Proxy AProxy AEP2EP3EP2Leaf 3Proxy BProxy BEP3EP4EP4EP2Proxy ALeaf 4Leaf 6EP3EP4COOPCOOPSingle BGP ASNAPIC ClusterIPN 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC ClusterACI Multi-PodInter-Pod Data Plane47BRKDCN-2949EP2EP1Proxy AProxy B4Spin
76、e encapsulates traffic to local leaf EP2Leaf 4EP1Proxy A6If policy allows it,EP2 receives the packetVM1 sends traffic destined to remote EP211VTEP IPVTEP IPClassClass-IDIDTenant PacketTenant PacketVNIDVNIDPolicy and network information carried across Pods5EP1Pod1 L4Proxy B*Leaf learns remote EP1 loc
77、ation and enforces policyEP2e1/1EP2 unknown,traffic is encapsulated to the local Proxy A Spine VTEP(adding S_Class information)Proxy A*2EP1e1/3=VXLAN Encap/DecapSpine encapsulates traffic to remote Proxy B Spine VTEP3EP1Leaf 4EP2Proxy BEP1 EPGEP2 EPGCConfigured on APICIPN 2023 Cisco and/or its affil
78、iates.All rights reserved.Cisco Public#CiscoLiveAPIC ClusterACI Multi-Pod Inter-Pod Data Plane(2)48BRKDCN-2949EP2EP1Leaf learns remote VM2 location(no need to enforce policy)Proxy A9EP2Pod2 L47VM2 sends traffic back to remote VM1VM1 receives the packet10=VXLAN Encap/DecapProxy AProxy BEP1Pod1 L4Prox
79、y B*Leaf enforces policy in ingress and,if allowed,encapsulates traffic to remote Leaf node L48EP1 EPGEP2 EPGCConfigured on APICIPN 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC ClusterACI Multi-PodInter-Pod Data Plane(3)49BRKDCN-294911From this point EP1 to EP2 com
80、munication is encapsulated Leaf to Leaf(VTEP to VTEP)and policy always applied at the ingress leaf(applies to both L2 and L3 communication)EP2EP1*EP1Pod1 L4Proxy B*Proxy AEP2Pod2 L4*EP1e1/3=VXLAN Encap/DecapProxy AProxy BIPN 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLive
81、ACI Multi-PodUse of Multicast for Inter-Pod Layer 2 BUM Traffic50BRKDCN-2949IGMP Join for(*,GIPo1)BD1 GIPo1:BD1 GIPo1:225.1.1.128IPN1IPN2Ingress replication for BUM*traffic not supported with Multi-Pod PIM Bidir is the only validated and supported optionScalable:only a single(*,G)entry is created in
82、 the IPN for each BDFast-convergent:no requirement for data-driven multicast state creation A spine is elected authoritative for each Bridge Domain:Generates an IGMP Join on a specific link toward the IPNAlways sends/receives BUM traffic on that link BUM traffic originated in the local PodBUM traffi
83、c originated from a remote PodBUM:Broadcast,Unknown Unicast,MulticastSpine 1 elected authoritative for BD1 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC ClusterACI Multi-Pod51BRKDCN-2949Use of Multicast for Inter-Pod BUM TrafficEP2EP1*6VM2 receives the BUM frameVM1
84、in BD1 generates a BUM*frame115BUM frame is flooded along one of the trees associated to MG1BUM:Layer 2 Broadcast,Unknown Unicast,Multicast3Spine 2 is designated to send MG1 traffic toward the IPNIPN replicates traffic to all the PODs that joined MG1(optimized delivery to Pods)4BD1 has associated MG
85、1,traffic is flooded intra-Pod via one multi-destination tree2 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveAPIC ClusterIPN1IPN2IPN3IPN4EP2APIC ClusterIPN1IPN2IPN3IPN4EP2ACI Multi-Pod52BRKDCN-2949PIM Bidir for BUM Supported TopologiesCreate full-mesh connections between
86、 IPN devicesMore costly for geo-dispersed Pods,as it requires more links between sitesAlternatively,connect local IPN devices with a port-channel interface(for resiliency)In both cases,it is criticalcritical to ensure that the preferred path toward the RP from any IPN devices is not via a spineRecom
87、mendation is to increase the OSPF cost of the interfaces between IPN and spinesinterface Ethernet1/49.4description L3 Link to Pod1-Spine1mtu 9150encapsulation dot1q 4ip address 192.168.1.1/31ip ospf cost 100ip ospf network point-to-pointip router ospf IPN area 0.0.0.0ip pim sparse-modeip dhcp relay
88、address 10.1.0.2 ip dhcp relay address 10.1.0.3 Full Mesh between remote IPN devicesDirectly connect local IPN devicesIPN1IPN2e1/49 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveACI Multi-Pod53BRKDCN-2949RP Redundancy with PIM BidirPOD 1POD 2IPN1IPN2IPN3IPN4APIC ClusterI
89、n PIM Bidir,only one device functions as active RP for a given group at the same timeCan leverage the Phantom RP configuration for providing resiliencyThe RP role is active on the device announcing the most specific route for the RPs addressIn case of failure of the active RP device,the shared tree
90、is immediately rebuilt toward the standby RP based on routing convergence eventCan deploy multiple RPs,each active for a sub-range of multicast groupsip pim rp-address 192.168.100.2 group-list 225.0.0.0/15 bidirip pim rp-address 192.168.100.2 group-list 239.255.255.240/28 bidirinterface loopback1des
91、cription Phantom RPip address 192.168.100.1/30ip ospf network point-to-pointip router ospf IPN area 0.0.0.0ip pim sparse-modeActive RP(IPN1)ip pim rp-address 192.168.100.2 group-list 225.0.0.0/15 bidirip pim rp-address 192.168.100.2 group-list 239.255.255.240/28 bidirinterface loopback1description P
92、hantom RPip address 192.168.100.1/29ip ospf network point-to-pointip router ospf IPN area 0.0.0.0ip pim sparse-modeStandby RP(IPN3)Connecting to the External Layer 3 Domain 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveWANClientPEPEPEPEConnecting to WAN Edge devices at B
93、order Leaf nodesVRF-Lite hand-off for extending L3 multi-tenancy outside the ACI fabric Support for host route advertisement out of the ACI FabricL3OutL3OutBorder LeafsConnecting ACI to Layer 3 DomainTraditional L3Out on the BL Nodes55BRKDCN-2949 2023 Cisco and/or its affiliates.All rights reserved.
94、Cisco Public#CiscoLiveMPMP-BGP EVPNBGP EVPNPrefix+ColorPrefix+ColorSP CoreBorder Leafs connect to PE router in SP coreSingle BGP EVPN session for all VRFsACI BL is advertising EVPN type-5 routes with BGP color communityInfra SRInfra SR-MPLS MPLS L3OutL3OutBorder LeafsConnecting ACI to Layer 3 Domain
95、SR-MPLS Handoff56BRKDCN-2949BGPBGP-LULUDC-PEDC-PESR-HandoffVRF-1VRF-2VRF-n171.1.1.0/24ACI Release 5.0(1)2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveConnecting to the External L3 DomainLocal L3Outs preferred over L3Outs in remote pods57BRKDCN-2949Pod1Pod1Pod2Pod2IPNIPNW
96、ANWANL3OutL3OutL3OutL3Out10.1.1.0/2410.1.1.0/24 via BL11via BL12BL21BL21BL22BL22BL11BL11BL12BL1210.1.1.0/24 via BL21via BL22Route received in both pods with same metric/attributesBGP table10.1.1.0/24 via BL11 TEP(metric 3)*via BL12 TEP(metric 3)*via BL21 TEP(metric 33)via BL22 TEP(metric 33)Remote p
97、od TEP routes are redistributed into IS-IS with metric configured in ISIS ISIS Domain PolicyDomain PolicyBGP tiebreaker is next-hop metric(IGP metric)BGP table10.1.1.0/24 via BL21 TEP(metric 3)*via BL22 TEP(metric 3)*via BL11 TEP(metric 33)via BL12 TEP(metric 33)Best Practice 2023 Cisco and/or its a
98、ffiliates.All rights reserved.Cisco Public#CiscoLiveConnecting to the External L3 DomainRemote pod L3Out may be used if it has a better external metric58BRKDCN-2949Pod1Pod1Pod2Pod2IPNIPNWANWANL3OutL3OutL3OutL3Out10.1.1.0/2410.1.1.0/24 via BL11via BL12BL21BL21BL22BL22BL11BL11BL12BL1210.1.1.0/24 via B
99、L11via BL12BGP table10.1.1.0/24 via BL11(MED 5)*via BL12(MED 5)*via BL21(MED 13)via BL22(MED 13)BGP MED is set to OSPF metric when redistributed into fabricRoutes with lower MED are selected(MED attribute is evaluated before IGP metric)10.1.1.0/24 110/510.1.1.0/24 110/13Route received in both pods w
100、ith different metric/attributesBGP table10.1.1.0/24 via BL11(MED 5)via BL12(MED 5)*via BL21(MED 13)via BL22(MED 13)2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveConnecting Multi-Pod to the Layer 3 DomainWhat happens when there are more than two pods?59BRKDCN-2949Pod1Pod1
101、Pod2Pod2Pod3Pod3IPNIPNL3OutL3OutWANECMPBL11BL11BL12BL12BL21BL21BL22BL22BGP table10.1.1.0/24 via BL11 TEP(metric 3)*via BL12 TEP(metric 3)*via BL21 TEP(metric 33)via BL22 TEP(metric 33)10.1.1.0/24BGP table10.1.1.0/24 via BL21 TEP(metric 3)*via BL22 TEP(metric 3)*via BL11 TEP(metric 33)via BL12 TEP(me
102、tric 33)BGP table10.1.1.0/24 via BL11 TEP(metric 33)*via BL12 TEP(metric 33)*via BL21 TEP(metric 33)*via BL22 TEP(metric 33)*A pod does not need a dedicated L3Out.Flows to external destinations can use an L3Out in another podTraffic flows are load balanced across all remote podsPath through IPN is n
103、ot considered in routing decision 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveConnecting Multi-Pod to the Layer 3 DomainHow to prefer one remote pod over another?60BRKDCN-2949Pod1Pod1Pod2Pod2Pod3Pod3IPNIPNL3OutL3OutWANECMPBL11BL11BL12BL12BL21BL21BL22BL2210.1.1.0/24Exam
104、ple:Use BGP attributes to make one pod preferredset local preference 150BGP table10.1.1.0/24 via BL21 Local Pref 150*via BL22 Local Pref 150*via BL11 Local Pref 100via BL12 Local Pref 100BGP table10.1.1.0/24 via BL21 Local Pref 150*via BL22 Local Pref 150*via BL11 Local Pref 100via BL12 Local Pref 1
105、00But change will affect all pods!2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveConnecting Multi-Pod to the Layer 3 DomainHow to prefer one remote pod over another?61BRKDCN-2949Pod1Pod1Pod2Pod2Pod3Pod3IPNIPNL3OutL3OutWANECMPBL11BL11BL12BL12BL21BL21BL22BL2210.1.1.0/24L3Ou
106、tAdding a local L3out may be a better option 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveConnecting to the External L3 DomainInfluencing inbound path:Host route advertisement62BRKDCN-2949Pod1Pod1Pod2Pod2IPNIPNWANWANL3OutL3OutL3OutL3Out192.168.1.201192.168.1.202192.168.
107、1.201/32192.168.1.201/32192.168.1.202/32192.168.1.202/32Host route advertisement can be enabled per BDEndpoints local to the pod will be downloaded to local BLs as/32 host routes 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveEdge DCs with Multi-Pod architecture63BRKDCN-2
108、949.Edge DC#1ACI PODEdge DC#2ACI PODEdge DC#3ACI PODEdge DC#24ACI PODEdge DC#25ACI PODIPNIPNService provider transportIP/MPLS-LDP/SR-MPLS/SRv6 APIC controllers are needed only in some Pods Communication across Pods is typically through SR-MPLS L3out 25 Pods per fabric is supported starting 6.0(1)rel
109、ease Leaf scale per fabric remains same.2 Spines per Pod is supported Latency requirement remains same-50 msec RTT requirement across APIC clusters and between switches and APIC No need to enable PIM-Bidir in IPN if L2 extension across Pod is not required 2023 Cisco and/or its affiliates.All rights
110、reserved.Cisco Public#CiscoLiveHost Route Advertisement OverviewBridge domain settingBorder leaf switches download/32 routes for endpoints connected to the local podHost route withdrawn from border leaf if endpoint moves to another pod or times outL3Out route-maps can be used to filter(permit or den
111、y)BD subnet routes and host routes and host route ranges64BRKDCN-2949For YourReferenceNetwork Services Integration 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveACI Multi-PodDesign options66BRKDCN-2949For your referenceActive and Standby pair deployed across PodsNo issue
112、s with asymmetric flowsTypical options for an Active/Active DC use caseIndependent Active/Standby pairs deployed in separate PodsUse of Symmetric PBR to avoid the creation of asymmetric paths crossing different active FW nodesActive/Active FW cluster nodes stretched across Sites(single logical FW)Re
113、quires the ability of discovering the same MAC/IP info in separate sites at the same timeSupported from ACI release 3.2(4d)with the use of Service-Graph with PBRActiveStandbyIPNIPNClusterIPNIPNIPNIPNIPNIPNActive/StandbyActive/StandbyActiveStandby 2023 Cisco and/or its affiliates.All rights reserved.
114、Cisco Public#CiscoLiveIPNIPNACI Multi-Pod:Active/Active cluster across podsNorth-South Traffic Flow67BRKDCN-2949ActiveActiveSpines in Pod1Spines in Pod110.1.1.1 via Service Leaf in Pod1(preferred)10.1.1.1 via Service Leaf in Pod1(preferred)10.1.1.1 via Pod2Spines in Pod2Spines in Pod210.1.1.1 via Se
115、rvice Leaf in Pod2(preferred)10.1.1.1 via Service Leaf in Pod2(preferred)10.1.1.1 via Pod1Pod1Pod2L3 Mode Active/Active ClusterFirewall IP:10.1.1.1EPGWebEPGWebL3OutL3Out-Site1Site1L3OutL3Out-Site2Site2External EPGCompute leaf always applies the PBR policyCompute leaf always applies the PBR policyPro
116、viderConsumerEPGWebCExtEPG 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveIPNIPNACI Multi-Pod:Active/Active cluster across podsEast-West Traffic Flow(Intra-Pod)68BRKDCN-2949ActiveActiveSpines in Pod1Spines in Pod110.1.1.1 via Service Leaf in Pod1(preferred)10.1.1.1 via Se
117、rvice Leaf in Pod1(preferred)10.1.1.1 via Pod2Spines in Pod2Spines in Pod210.1.1.1 via Service Leaf in Pod2(preferred)10.1.1.1 via Service Leaf in Pod2(preferred)10.1.1.1 via Pod1Pod1Pod2L3 Mode Active/Active ClusterFirewall IP:10.1.1.1EPGWebEPGAppEPGWebEPGAppProviderConsumerEPGAppEPGWebC 2023 Cisco
118、 and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveIPNIPNACI Multi-Pod:Active/Active cluster across podsEast-West Traffic Flow(Inter-Pod)incoming traffic69BRKDCN-2949ActiveActiveSpines in Pod1Spines in Pod110.1.1.1 via Service Leaf in Pod1(preferred)10.1.1.1 via Service Leaf in Pod1(pr
119、eferred)10.1.1.1 via Pod2Pod1Pod2L3 Mode Active/Active ClusterFirewall IP:10.1.1.1EPGWebEPGAppProviderConsumerEPGAppEPGWebC 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveIPNIPNACI Multi-Pod:Active/Active cluster across podsEast-West Traffic Flow(Inter-Pod)return traffic7
120、0BRKDCN-2949ActiveActiveSpines in Pod2Spines in Pod210.1.1.1 via Service Leaf in Pod2(preferred)10.1.1.1 via Service Leaf in Pod2(preferred)10.1.1.1 via Pod1Pod1Pod2L3 Mode Active/Active ClusterFirewall IP:10.1.1.1EPGWebEPGAppEven if asymmetric redirection happens,ASA/FTD clustering ensures traffic
121、is forwarded to the same firewall via control link.ProviderConsumerEPGAppEPGWebCMulti-Pod with Remote Leaf 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveACI Remote Leaf with Multi-PodDirect Forwarding between RL Pairs Part of the Same Pod IP NetworkRL Pair 1RL Pair 1Pod1
122、Pod172BRKDCN-2949RL Pair 2RL Pair 2Pod1Pod1ACI FabricACI FabricPod1Pod1ACI FabricACI FabricPod2Pod2RL Pair 1 associated to Pod1RL Pair 2 associated to Pod1Data plane traffic is forwarded directly between RLs.Does not need to go to spinePod1 Control PlanePod1 Control PlaneData PlaneData Plane 2023 Ci
123、sco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveACI Remote Leaf with Multi-PodDirect Forwarding between RL Pairs Part of Different Pods IP NetworkACI FabricACI FabricPod1Pod1ACI FabricACI FabricPod2Pod273BRKDCN-2949RL Pair 2RL Pair 2Pod2Pod2RL Pair 1Pod1RL Pair 1 associated to Po
124、d1RL Pair 2 associated to Pod2Data plane traffic is forwarded directly between RLs.Does not need to go to spinePod1 Control PlanePod1 Control PlaneData PlaneData PlanePod2 Control PlanePod2 Control Plane 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveACI Remote Physical L
125、eafRL Pair Resiliency in a Pod Failure ScenarioIP NetworkRL PairRL PairPod1Pod174BRKDCN-2949XRL Pair 1 associated to Pod1After Pod1s failure,RL pair 1 dynamically associates to Pod2 ACI FabricACI FabricPod1Pod1ACI FabricACI FabricPod2Pod2Pod1 Control PlanePod1 Control PlanePod2 Control PlanePod2 Con
126、trol Plane 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLiveUseful Links75BRKDCN-2949ACI Multi-Pod White Paperhttp:/ Multi-Pod Configuration Paperhttps:/ Multi-Pod and Service Node Integration White Paperhttps:/ Remote Leaf Architecture White Paperhttps:/ 2023 Cisco and/or
127、its affiliates.All rights reserved.Cisco Public#CiscoLiveFill out your session surveys!Attendees who fill out a minimum of four session surveys and the overall event survey will get Cisco Live-branded socks(while supplies last)!76BRKDCN-2949These points help you get on the leaderboard and increase y
128、our chances of winning daily and grand prizesAttendees will also earn 100 points in the Cisco Live Challenge for every survey completed.2023 Cisco and/or its affiliates.All rights reserved.Cisco PublicContinue your educationVisit the Cisco Showcase for related demosBook your one-on-oneMeet the Engin
129、eer meetingAttend the interactive education with DevNet,Capture the Flag,and Walk-in LabsVisit the On-Demand Library for more sessions at www.CiscoL you#CiscoLive 2023 Cisco and/or its affiliates.All rights reserved.Cisco Public#CiscoLive79Gamify your Cisco Live experience!Get points Get points for
130、attending this session!for attending this session!Open the Cisco Events App.Click on Cisco Live Challenge in the side menu.Click on View Your Badges at the top.Click the+at the bottom of the screen and scan the QR code:How:123479 2023 Cisco and/or its affiliates.All rights reserved.Cisco PublicBRKDCN-2949#CiscoLive