Juniper EVPN-VXLAN & DCI
Goal
In this tutorial we are going to setup 2 sites: Site A and site B and connect both sites together using a MPLS backbone. Both sites will be configured using EVPN-VXLAN centrally Routed Bridging (CRB).
Prerequisites
For this tutorial I will be using :
- EVE-NG 2.0.3-112
- VQFX model: vqfx-10000 running JUNOS 19.4R1.10 for spines and leaves
- VMX running Junos: 18.2R1.9 for the core routers
- Debian VM's servers
Diagram
Setup and configuration
Site A
| Devices | Role | lo.0 IP | xe-0/0/0 IP | xe-0/0/1 IP | xe-0/0/2 IP | xe-0/0/9 IP | mgmt | AS | local-as |
| qfxssw27 | spine1 | 10.179.0.1/32 | 172.16.1.49/30 | 172.16.2.2/30 | 172.16.3.1/30 | 10.193.0.78/24 | 64601 | 65030 | |
| qfxssw28 | spine2 | 10.179.0.2/32 | 172.16.1.6/30 | 172.16.2.6/30 | 172.16.3.5/30 | 10.193.0.79/24 | 64601 | 65031 | |
| qfxlsw27 | leaf1 | 10.179.0.5/32 | 172.16.1.50/30 | 172.16.1.5/30 | 10.193.0.80/24 | 64601 | 65032 | ||
| qfxlsw28 | leaf2 | 10.179.0.6/32 | 172.16.2.1/30 | 172.16.2.5/30 | 10.193.0.81/24 | 64601 | 65033 | ||
| cr1 | core router | 10.179.0.7/32 | 172.16.3.2/30 | 172.16.3.6/30 | 172.16.4.0/31 | 10.193.0.86/24 | 100 |
| vlan | vlan-id | vni | Network | irb IP | VG |
| private1-a-dal | 2005 | 102005 | 10.192.64.0/22 | 10.192.64.2/22 | 10.192.64.1 |
| private1-b-dal | 2006 | 102006 | 10.192.80.0/22 | 10.192.80.2/22 | 10.192.80.1 |
Spine1 configuration
set system host-name qfxssw27-dfw set interfaces xe-0/0/0 description link_leaf1 set interfaces xe-0/0/0 unit 0 family inet address 172.16.1.49/30 set interfaces xe-0/0/1 description link_leaf2 set interfaces xe-0/0/1 unit 0 family inet address 172.16.2.2/30 set interfaces xe-0/0/2 description link_cr1 set interfaces xe-0/0/2 unit 0 family inet address 172.16.3.1/30 set interfaces em0 unit 0 family inet address 10.193.0.78/24 set interfaces irb unit 2006 proxy-macip-advertisement set interfaces irb unit 2006 virtual-gateway-accept-data set interfaces irb unit 2006 family inet address 10.192.80.2/22 virtual-gateway-address 10.192.80.1 set interfaces irb unit 3502 proxy-macip-advertisement set interfaces irb unit 3502 virtual-gateway-accept-data set interfaces irb unit 3502 family inet address 10.192.64.2/22 virtual-gateway-address 10.192.64.1 set interfaces lo0 unit 0 family inet address 10.179.0.1/32 primary set policy-options prefix-list fab-lo0s 10.179.0.0/24 set policy-options policy-statement EVPN-IMPORT term VNI102005 from community com2006 set policy-options policy-statement EVPN-IMPORT term VNI102005 then accept set policy-options policy-statement EVPN-IMPORT2006 term 1 from community com2005 set policy-options policy-statement EVPN-IMPORT2006 term 1 then accept set policy-options policy-statement PFE-ECMP then load-balance per-packet set policy-options policy-statement export-dc-routes term fabric from protocol static set policy-options policy-statement export-dc-routes term fabric from prefix-list-filter fab-lo0s exact set policy-options policy-statement export-dc-routes term fabric then accept set policy-options policy-statement export-dc-routes then reject set policy-options policy-statement import-overlay term 1 from family evpn set policy-options policy-statement import-overlay term 1 from next-hop 10.179.1.1 set policy-options policy-statement import-overlay term 1 from nlri-route-type 1 set policy-options policy-statement import-overlay term 1 from nlri-route-type 2 set policy-options policy-statement import-overlay term 1 then reject set policy-options policy-statement import-overlay then accept set policy-options policy-statement send-direct term send-lo0 from protocol direct set policy-options policy-statement send-direct term send-lo0 from interface lo0.0 set policy-options policy-statement send-direct term send-lo0 then accept set policy-options policy-statement send-direct term 2 from protocol bgp set policy-options policy-statement send-direct term 2 from route-filter 0.0.0.0/0 prefix-length-range /32-/32 set policy-options policy-statement send-direct term 2 then accept set policy-options community com2005 members target:64601:2 set policy-options community com2006 members target:64601:3 set routing-instances vlan2005 routing-options auto-export set routing-instances vlan2005 instance-type vrf set routing-instances vlan2005 interface irb.3502 set routing-instances vlan2005 route-distinguisher 10.179.0.1:2 set routing-instances vlan2005 vrf-import EVPN-IMPORT set routing-instances vlan2005 vrf-target target:64601:2 set routing-instances vlan2006 routing-options auto-export set routing-instances vlan2006 instance-type vrf set routing-instances vlan2006 interface irb.2006 set routing-instances vlan2006 route-distinguisher 10.179.0.1:3 set routing-instances vlan2006 vrf-import EVPN-IMPORT2006 set routing-instances vlan2006 vrf-target target:64601:3 set routing-options static route 0.0.0.0/0 next-hop 10.193.0.1 set routing-options static route 0.0.0.0/0 no-readvertise set routing-options static route 10.179.0.0/24 discard set routing-options forwarding-table export PFE-ECMP set routing-options forwarding-table ecmp-fast-reroute set routing-options router-id 10.179.0.1 set routing-options autonomous-system 64601 set protocols evpn vni-options vni 102005 vrf-target target:1:2005 set protocols evpn vni-options vni 102006 vrf-target target:1:2006 set protocols evpn encapsulation vxlan set protocols evpn multicast-mode ingress-replication set protocols evpn default-gateway no-gateway-community set protocols evpn extended-vni-list all set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY mtu-discovery set protocols bgp group UNDERLAY export send-direct set protocols bgp group UNDERLAY local-as 65030 set protocols bgp group UNDERLAY multipath multiple-as set protocols bgp group UNDERLAY bfd-liveness-detection minimum-interval 1000 set protocols bgp group UNDERLAY bfd-liveness-detection multiplier 3 set protocols bgp group UNDERLAY bfd-liveness-detection session-mode automatic set protocols bgp group UNDERLAY neighbor 172.16.2.1 description leaf2 set protocols bgp group UNDERLAY neighbor 172.16.2.1 peer-as 65033 set protocols bgp group UNDERLAY neighbor 172.16.1.50 description leaf1 set protocols bgp group UNDERLAY neighbor 172.16.1.50 peer-as 65032 set protocols bgp group EVPN-IBGP type internal set protocols bgp group EVPN-IBGP local-address 10.179.0.1 set protocols bgp group EVPN-IBGP import import-overlay set protocols bgp group EVPN-IBGP family evpn signaling set protocols bgp group EVPN-IBGP cluster 1.1.1.1 set protocols bgp group EVPN-IBGP multipath set protocols bgp group EVPN-IBGP bfd-liveness-detection minimum-interval 350 set protocols bgp group EVPN-IBGP bfd-liveness-detection multiplier 3 set protocols bgp group EVPN-IBGP bfd-liveness-detection session-mode automatic set protocols bgp group EVPN-IBGP neighbor 10.179.0.5 description lfsw01 set protocols bgp group EVPN-IBGP neighbor 10.179.0.6 description lfsw02 set protocols bgp group EVPN-IBGP neighbor 10.179.0.2 description spine2 set protocols bgp group EVPN-IBGP neighbor 10.179.1.1 description dc2_spine1 set protocols bgp group core type external set protocols bgp group core export export-dc-routes set protocols bgp group core peer-as 100 set protocols bgp group core local-as 65030 set protocols bgp group core neighbor 172.16.3.2 set protocols igmp-snooping vlan default set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 10.179.0.1:1 set switch-options vrf-target target:64512:1111 set vlans default vlan-id 1 set vlans private1-a-dal vlan-id 2005 set vlans private1-a-dal l3-interface irb.3502 set vlans private1-a-dal vxlan vni 102005 set vlans private1-b-dal vlan-id 2006 set vlans private1-b-dal l3-interface irb.2006 set vlans private1-b-dal vxlan vni 102006
Spine2 configuration
set system host-name qfxssw28-dfw set interfaces xe-0/0/0 description link_leaf1 set interfaces xe-0/0/0 unit 0 family inet address 172.16.1.6/30 set interfaces xe-0/0/1 description link_leaf2 set interfaces xe-0/0/1 unit 0 family inet address 172.16.2.6/30 set interfaces xe-0/0/2 description link_cr2 set interfaces xe-0/0/2 unit 0 family inet address 172.16.3.5/30 set interfaces em0 unit 0 family inet address 10.193.0.79/24 set interfaces irb unit 2006 proxy-macip-advertisement set interfaces irb unit 2006 virtual-gateway-accept-data set interfaces irb unit 2006 family inet address 10.192.80.3/22 virtual-gateway-address 10.192.80.1 set interfaces irb unit 3502 proxy-macip-advertisement set interfaces irb unit 3502 virtual-gateway-accept-data set interfaces irb unit 3502 family inet address 10.192.64.3/22 virtual-gateway-address 10.192.64.1 set interfaces lo0 unit 0 family inet address 10.179.0.2/32 primary set policy-options prefix-list fab-lo0s 10.179.0.0/24 set policy-options policy-statement EVPN-IMPORT term VNI102005 from community com2006 set policy-options policy-statement EVPN-IMPORT term VNI102005 then accept set policy-options policy-statement EVPN-IMPORT2006 term 1 from community com2005 set policy-options policy-statement EVPN-IMPORT2006 term 1 then accept set policy-options policy-statement PFE-ECMP then load-balance per-packet set policy-options policy-statement export-dc-routes term fabric from protocol static set policy-options policy-statement export-dc-routes term fabric from prefix-list-filter fab-lo0s exact set policy-options policy-statement export-dc-routes term fabric then accept set policy-options policy-statement export-dc-routes then reject set policy-options policy-statement import-overlay term 1 from family evpn set policy-options policy-statement import-overlay term 1 from next-hop 10.179.1.1 set policy-options policy-statement import-overlay term 1 from nlri-route-type 1 set policy-options policy-statement import-overlay term 1 from nlri-route-type 2 set policy-options policy-statement import-overlay term 1 then reject set policy-options policy-statement import-overlay then accept set policy-options policy-statement send-direct term send-lo0 from protocol direct set policy-options policy-statement send-direct term send-lo0 from interface lo0.0 set policy-options policy-statement send-direct term send-lo0 then accept set policy-options policy-statement send-direct term 2 from protocol bgp set policy-options policy-statement send-direct term 2 from route-filter 0.0.0.0/0 prefix-length-range /32-/32 set policy-options policy-statement send-direct term 2 then accept set policy-options community com2005 members target:64601:2 set policy-options community com2006 members target:64601:3 set routing-instances vlan2005 routing-options auto-export set routing-instances vlan2005 instance-type vrf set routing-instances vlan2005 interface irb.3502 set routing-instances vlan2005 route-distinguisher 10.179.0.1:2 set routing-instances vlan2005 vrf-import EVPN-IMPORT set routing-instances vlan2005 vrf-target target:64601:2 set routing-instances vlan2006 routing-options auto-export set routing-instances vlan2006 instance-type vrf set routing-instances vlan2006 interface irb.2006 set routing-instances vlan2006 route-distinguisher 10.179.0.1:3 set routing-instances vlan2006 vrf-import EVPN-IMPORT2006 set routing-instances vlan2006 vrf-target target:64601:3 set routing-options static route 0.0.0.0/0 next-hop 10.193.0.1 set routing-options static route 0.0.0.0/0 no-readvertise set routing-options static route 10.179.0.0/24 discard set routing-options forwarding-table export PFE-ECMP set routing-options forwarding-table ecmp-fast-reroute set routing-options router-id 10.179.0.2 set routing-options autonomous-system 64601 set protocols evpn vni-options vni 102005 vrf-target target:1:2005 set protocols evpn vni-options vni 102006 vrf-target target:1:2006 set protocols evpn encapsulation vxlan set protocols evpn multicast-mode ingress-replication set protocols evpn default-gateway no-gateway-community set protocols evpn extended-vni-list all set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY mtu-discovery set protocols bgp group UNDERLAY export send-direct set protocols bgp group UNDERLAY local-as 65031 set protocols bgp group UNDERLAY multipath multiple-as set protocols bgp group UNDERLAY bfd-liveness-detection minimum-interval 1000 set protocols bgp group UNDERLAY bfd-liveness-detection multiplier 3 set protocols bgp group UNDERLAY bfd-liveness-detection session-mode automatic set protocols bgp group UNDERLAY neighbor 172.16.1.5 description leaf1 set protocols bgp group UNDERLAY neighbor 172.16.1.5 peer-as 65032 set protocols bgp group UNDERLAY neighbor 172.16.2.5 description leaf2 set protocols bgp group UNDERLAY neighbor 172.16.2.5 peer-as 65033 set protocols bgp group EVPN-IBGP type internal set protocols bgp group EVPN-IBGP local-address 10.179.0.2 set protocols bgp group EVPN-IBGP import import-overlay set protocols bgp group EVPN-IBGP family evpn signaling set protocols bgp group EVPN-IBGP cluster 2.2.2.2 set protocols bgp group EVPN-IBGP multipath set protocols bgp group EVPN-IBGP bfd-liveness-detection minimum-interval 350 set protocols bgp group EVPN-IBGP bfd-liveness-detection multiplier 3 set protocols bgp group EVPN-IBGP bfd-liveness-detection session-mode automatic set protocols bgp group EVPN-IBGP neighbor 10.179.0.5 description lfsw01 set protocols bgp group EVPN-IBGP neighbor 10.179.0.6 description lfsw02 set protocols bgp group EVPN-IBGP neighbor 10.179.0.1 description spine1 set protocols bgp group EVPN-IBGP neighbor 10.179.1.1 description dc2_spine1 set protocols bgp group EVPN-IBGP neighbor 10.179.1.2 description dc2_spine2 set protocols bgp group core type external set protocols bgp group core export export-dc-routes set protocols bgp group core peer-as 100 set protocols bgp group core local-as 65031 set protocols bgp group core neighbor 172.16.3.6 set protocols igmp-snooping vlan default set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 10.179.0.2:1 set switch-options vrf-target target:64512:1111 set vlans default vlan-id 1 set vlans private1-a-dal vlan-id 2005 set vlans private1-a-dal l3-interface irb.3502 set vlans private1-a-dal vxlan vni 102005 set vlans private1-b-dal vlan-id 2006 set vlans private1-b-dal l3-interface irb.2006 set vlans private1-b-dal vxlan vni 102006
Leaf1 configuration
set system host-name qfxlsw27-dfw set interfaces xe-0/0/0 description link_spine1 set interfaces xe-0/0/0 unit 0 family inet address 172.16.1.50/30 set interfaces xe-0/0/1 description link_spine2 set interfaces xe-0/0/1 unit 0 family inet address 172.16.1.5/30 set interfaces xe-0/0/10 description srv101 set interfaces xe-0/0/10 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/10 unit 0 family ethernet-switching vlan members private1-b-dal set interfaces xe-0/0/11 description srv100 set interfaces xe-0/0/11 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/11 unit 0 family ethernet-switching vlan members private1-a-dal set interfaces em0 unit 0 family inet address 10.193.0.80/24 set interfaces lo0 unit 0 family inet address 10.179.0.5/32 set policy-options policy-statement PFE-ECMP then load-balance per-packet set policy-options policy-statement send-direct term send-lo0 from protocol direct set policy-options policy-statement send-direct term send-lo0 then accept set routing-options static route 0.0.0.0/0 next-hop 10.193.0.1 set routing-options static route 0.0.0.0/0 no-readvertise set routing-options forwarding-table export PFE-ECMP set routing-options forwarding-table ecmp-fast-reroute set routing-options router-id 10.179.0.5 set routing-options autonomous-system 64601 set protocols evpn vni-options vni 102005 vrf-target target:1:2005 set protocols evpn vni-options vni 102006 vrf-target target:1:2006 set protocols evpn encapsulation vxlan set protocols evpn multicast-mode ingress-replication set protocols evpn extended-vni-list all set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY export send-direct set protocols bgp group UNDERLAY local-as 65032 set protocols bgp group UNDERLAY multipath multiple-as set protocols bgp group UNDERLAY bfd-liveness-detection minimum-interval 1000 set protocols bgp group UNDERLAY bfd-liveness-detection multiplier 3 set protocols bgp group UNDERLAY bfd-liveness-detection session-mode automatic set protocols bgp group UNDERLAY neighbor 172.16.1.6 description spine2 set protocols bgp group UNDERLAY neighbor 172.16.1.6 peer-as 65031 set protocols bgp group UNDERLAY neighbor 172.16.1.49 description spine1 set protocols bgp group UNDERLAY neighbor 172.16.1.49 peer-as 65030 set protocols bgp group EVPN-IBGP type internal set protocols bgp group EVPN-IBGP local-address 10.179.0.5 set protocols bgp group EVPN-IBGP family evpn signaling set protocols bgp group EVPN-IBGP multipath set protocols bgp group EVPN-IBGP bfd-liveness-detection minimum-interval 350 set protocols bgp group EVPN-IBGP bfd-liveness-detection multiplier 3 set protocols bgp group EVPN-IBGP bfd-liveness-detection session-mode automatic set protocols bgp group EVPN-IBGP neighbor 10.179.0.1 description spsw01 set protocols bgp group EVPN-IBGP neighbor 10.179.0.2 description spsw02 set protocols lldp disable set protocols lldp port-id-subtype interface-name set protocols lldp interface all set protocols lldp-med interface all set protocols igmp-snooping vlan default set switch-options service-id 1 set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 10.179.0.5:1 set switch-options vrf-target target:64512:1111 set vlans private1-a-dal vlan-id 2005 set vlans private1-a-dal vxlan vni 102005 set vlans private1-b-dal vlan-id 2006 set vlans private1-b-dal vxlan vni 102006
Leaf2 configuration
set system host-name qfxlsw28 set interfaces xe-0/0/0 description link_spine1 set interfaces xe-0/0/0 unit 0 family inet address 172.16.2.1/30 set interfaces xe-0/0/1 description link_spine2 set interfaces xe-0/0/1 unit 0 family inet address 172.16.2.5/30 set interfaces xe-0/0/10 description srv104 set interfaces xe-0/0/10 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/10 unit 0 family ethernet-switching vlan members private1-b-dal set interfaces xe-0/0/11 description srv103 set interfaces xe-0/0/11 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/11 unit 0 family ethernet-switching vlan members private1-a-dal set interfaces em0 unit 0 family inet address 10.193.0.81/24 set interfaces lo0 unit 0 family inet address 10.179.0.6/32 set policy-options policy-statement PFE-ECMP then load-balance per-packet set policy-options policy-statement send-direct term send-lo0 from protocol direct set policy-options policy-statement send-direct term send-lo0 then accept set routing-options static route 0.0.0.0/0 next-hop 10.193.0.1 set routing-options static route 0.0.0.0/0 no-readvertise set routing-options forwarding-table export PFE-ECMP set routing-options forwarding-table ecmp-fast-reroute set routing-options router-id 10.179.0.6 set routing-options autonomous-system 64601 set protocols evpn vni-options vni 102005 vrf-target target:1:2005 set protocols evpn vni-options vni 102006 vrf-target target:1:2006 set protocols evpn encapsulation vxlan set protocols evpn multicast-mode ingress-replication set protocols evpn extended-vni-list all set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY export send-direct set protocols bgp group UNDERLAY local-as 65033 set protocols bgp group UNDERLAY multipath multiple-as set protocols bgp group UNDERLAY bfd-liveness-detection minimum-interval 1000 set protocols bgp group UNDERLAY bfd-liveness-detection multiplier 3 set protocols bgp group UNDERLAY bfd-liveness-detection session-mode automatic set protocols bgp group UNDERLAY neighbor 172.16.2.2 description spine1 set protocols bgp group UNDERLAY neighbor 172.16.2.2 peer-as 65030 set protocols bgp group UNDERLAY neighbor 172.16.2.6 description spine2 set protocols bgp group UNDERLAY neighbor 172.16.2.6 peer-as 65031 set protocols bgp group EVPN-IBGP type internal set protocols bgp group EVPN-IBGP local-address 10.179.0.6 set protocols bgp group EVPN-IBGP family evpn signaling set protocols bgp group EVPN-IBGP multipath set protocols bgp group EVPN-IBGP bfd-liveness-detection minimum-interval 350 set protocols bgp group EVPN-IBGP bfd-liveness-detection multiplier 3 set protocols bgp group EVPN-IBGP bfd-liveness-detection session-mode automatic set protocols bgp group EVPN-IBGP neighbor 10.179.0.1 description spsw01 set protocols bgp group EVPN-IBGP neighbor 10.179.0.2 description spsw02 set protocols lldp disable set protocols lldp port-id-subtype interface-name set protocols lldp interface all set protocols lldp-med interface all set protocols igmp-snooping vlan default set switch-options service-id 1 set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 10.179.0.6:1 set switch-options vrf-target target:64512:1111 set vlans default vlan-id 1 set vlans private1-a-dal vlan-id 2005 set vlans private1-a-dal vxlan vni 102005 set vlans private1-b-dal vlan-id 2006 set vlans private1-b-dal vxlan vni 102006
Site A EVPN/VXLAN testing & Verification
Underlay Verification
We are performing the test only on leaf1 and spine1, the same result will be also true for leaf2 and spine2. To verify this we will power off spine 1 later in this tutorial and make sure that spine2 will be taking all the traffic.
- on leaf1
root@qfxlsw27-dfw> show bgp summary group UNDERLAY
Threading mode: BGP I/O
Groups: 2 Peers: 4 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0 42 36 0 0 0 0
inet.0
6 6 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.6 65031 1844 1853 0 35 14:01:33 Establ
inet.0: 4/4/4/0
172.16.1.49 65030 463 463 0 35 3:29:50 Establ
inet.0: 2/2/2/0
We have BGP session with spine1(1.49) and spein2 (1.6) up.
root@qfxlsw27-dfw> show route advertising-protocol bgp 172.16.1.49 inet.0: 15 destinations, 16 routes (15 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.0.2/32 Self 65031 I * 10.179.0.5/32 Self I * 10.179.0.6/32 Self 65031 65033 I
root@qfxlsw27-dfw> show route advertising-protocol bgp 172.16.1.6 inet.0: 15 destinations, 16 routes (15 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.0.1/32 Self 65030 I * 10.179.0.5/32 Self I
root@qfxlsw27-dfw> show route receive-protocol bgp 172.16.1.49 table inet.0 inet.0: 15 destinations, 16 routes (15 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.0.1/32 172.16.1.49 65030 I 10.179.0.2/32 172.16.1.49 65030 65033 65031 I 10.179.0.6/32 172.16.1.49 65030 65033 I
root@qfxlsw27-dfw> show route receive-protocol bgp 172.16.1.6 table inet.0 inet.0: 15 destinations, 16 routes (15 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path 10.179.0.1/32 172.16.1.6 65031 65033 65030 I * 10.179.0.2/32 172.16.1.6 65031 I * 10.179.0.6/32 172.16.1.6 65031 65033 I
-On spine1
root@qfxssw27-dfw> show bgp summary group UNDERLAY
Threading mode: BGP I/O
Groups: 3 Peers: 7 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0 83 56 0 0 0 0
inet.0
15 7 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.50 65032 443 437 0 1 3:17:55 Establ
inet.0: 3/7/7/0
172.16.2.1 65033 1499 1493 0 1 11:19:41 Establ
inet.0: 3/7/7/0
root@qfxssw27-dfw> show route receive-protocol bgp 172.16.1.50 table inet.0 inet.0: 19 destinations, 28 routes (19 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path 10.179.0.2/32 172.16.1.50 65032 65031 I * 10.179.0.5/32 172.16.1.50 65032 I 10.179.0.6/32 172.16.1.50 65032 65031 65033 I
root@qfxssw27-dfw> show route receive-protocol bgp 172.16.2.1 table inet.0 inet.0: 19 destinations, 26 routes (19 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.0.2/32 172.16.2.1 65033 65031 I 10.179.0.5/32 172.16.2.1 65033 65031 65032 I * 10.179.0.6/32 172.16.2.1 65033 I root@qfxssw27-dfw> show route advertising-protocol bgp 172.16.1.50 inet.0: 19 destinations, 26 routes (19 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.0.1/32 Self I * 10.179.0.2/32 Self 65033 65031 I * 10.179.0.6/32 Self 65033 I
root@qfxssw27-dfw> show route advertising-protocol bgp 172.16.2.1 inet.0: 19 destinations, 26 routes (19 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.0.1/32 Self I * 10.179.0.5/32 Self 65032 I
Now that we know that leaves and spines can reach each other loopback, let us now verify the Overlay network.
Overlay Verification
- On leaf1
root@qfxlsw27-dfw> show bgp summary group EVPN-IBGP
Threading mode: BGP I/O
Groups: 2 Peers: 4 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
72 36 0 0 0 0
inet.0
6 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.179.0.1 64601 3064 3030 0 3 22:55:21 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 24/36/36/0
default-switch.evpn.0: 24/36/36/0
10.179.0.2 64601 9817 9625 0 4 3d 0:54:21 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 12/36/36/0
default-switch.evpn.0: 12/36/36/0
- On spine1
root@qfxssw27-dfw# run show bgp summary group EVPN-IBGP
Threading mode: BGP I/O
Groups: 3 Peers: 7 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
83 56 0 0 0 0
inet.0
15 7 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn
State|#Active/Received/Accepted/Damped...
10.179.0.2 64601 5763 5764 0 0 1d 19:46:04 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 12/24/24/0
default-switch.evpn.0: 12/24/24/0
10.179.0.5 64601 6253 6262 0 0 1d 23:20:01 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 6/6/6/0
default-switch.evpn.0: 6/6/6/0
10.179.0.6 64601 6253 6302 0 0 1d 23:19:57 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 6/6/6/0
default-switch.evpn.0: 6/6/6/0
We can see that on leaf1 we are able to see spine1 an spine2 loopback and on spine1 we are able to see leaf1, leaf2 and spine2 loopback.
Connectivity test
Test that srv100 with IP address 10.192.64.100 can ping 10.192.64.200 and 10.192.80.200 on leaf2.
Now that we know everything is working in site A, it is time to move to site B
Site B
| Devices | Role | lo.0 IP | xe-0/0/0 IP | xe-0/0/1 IP | xe-0/0/2 IP | xe-0/0/9 IP | mgmt | AS | local-as |
| qfxssw29 | spine1 | 10.179.1.1/32 | 172.16.1.41/30 | 172.16.1.33/30 | 172.16.5.1/30 | 10.193.0.82/24 | 64601 | 65034 | |
| qfxssw30 | spine2 | 10.179.1.2/32 | 172.16.1.45/30 | 172.16.1.37/30 | 172.16.5.5/30 | 10.193.0.83/24 | 64601 | 65035 | |
| qfxlsw29 | leaf1 | 10.179.1.5/32 | 172.16.1.42/30 | 172.16.1.46/30 | 10.193.0.84/24 | 64601 | 65036 | ||
| qfxlsw30 | leaf2 | 10.179.1.6/32 | 172.16.1.34/30 | 172.16.1.38/30 | 10.193.0.85/24 | 64601 | 65037 | ||
| cr2 | core router | 10.179.1.7/32 | 172.16.5.2/30 | 172.16.5.6/30 | 172.16.6.0/31 | 10.193.0.87/24 | 100 |
| vlan | vlan-id | vni | Network | irb IP | VG |
| private1-a-dal | 2005 | 102005 | 10.192.64.0/22 | 10.192.64.3/22 | 10.192.64.1 |
| private1-a-sfo | 2007 | 102007 | 10.192.96.0/22 | 10.192.96.2/22 | 10.192.96.1 |
| private1-b-sfo | 2008 | 102008 | 10.192.112.0/22 | 10.192.112.2/22 | 10.192.112.1 |
Spine1 configuration
set system host-name qfxssw29-dfw set interfaces xe-0/0/0 description link_leaf1 set interfaces xe-0/0/0 unit 0 family inet address 172.16.1.41/30 set interfaces xe-0/0/1 description link_leaf2 set interfaces xe-0/0/1 unit 0 family inet address 172.16.1.33/30 set interfaces xe-0/0/2 description link_cr2 set interfaces xe-0/0/2 unit 0 family inet address 172.16.5.1/30 set interfaces em0 unit 0 family inet address 10.193.0.82/24 set interfaces irb unit 2007 proxy-macip-advertisement set interfaces irb unit 2007 virtual-gateway-accept-data set interfaces irb unit 2007 family inet address 10.192.96.2/22 virtual-gateway-address 10.192.96.1 set interfaces irb unit 2008 proxy-macip-advertisement set interfaces irb unit 2008 virtual-gateway-accept-data set interfaces irb unit 2008 family inet address 10.192.112.2/22 virtual-gateway-address 10.192.112.1 set interfaces irb unit 3502 proxy-macip-advertisement set interfaces irb unit 3502 virtual-gateway-accept-data set interfaces irb unit 3502 family inet address 10.192.64.4/22 virtual-gateway-address 10.192.64.1 set interfaces lo0 unit 0 family inet address 10.179.1.1/32 primary set policy-options prefix-list fab-lo0s 10.179.1.0/24 set policy-options policy-statement EVPN-IMPORT term VNI102007 from community com2008 set policy-options policy-statement EVPN-IMPORT term VNI102007 then accept set policy-options policy-statement EVPN-IMPORT term VNI102005 from community com2005 set policy-options policy-statement EVPN-IMPORT term VNI102005 then accept set policy-options policy-statement EVPN-IMPORT2008 term 1 from community com2007 set policy-options policy-statement EVPN-IMPORT2008 term 1 then accept set policy-options policy-statement PFE-ECMP then load-balance per-packet set policy-options policy-statement export-dc-routes term fabric from protocol static set policy-options policy-statement export-dc-routes term fabric from prefix-list-filter fab-lo0s exact set policy-options policy-statement export-dc-routes term fabric then accept set policy-options policy-statement export-dc-routes then reject set policy-options policy-statement import-overlay term 1 from family evpn set policy-options policy-statement import-overlay term 1 from next-hop 10.179.0.1 set policy-options policy-statement import-overlay term 1 from nlri-route-type 1 set policy-options policy-statement import-overlay term 1 from nlri-route-type 2 set policy-options policy-statement import-overlay term 1 then reject set policy-options policy-statement import-overlay term 2 from family evpn set policy-options policy-statement import-overlay term 2 from next-hop 10.179.0.2 set policy-options policy-statement import-overlay term 2 from nlri-route-type 1 set policy-options policy-statement import-overlay term 2 from nlri-route-type 2 set policy-options policy-statement import-overlay term 2 then reject set policy-options policy-statement import-overlay then accept set policy-options policy-statement send-direct term send-lo0 from protocol direct set policy-options policy-statement send-direct term send-lo0 from interface lo0.0 set policy-options policy-statement send-direct term send-lo0 then accept set policy-options community com2005 members target:65001:4 set policy-options community com2007 members target:65001:2 set policy-options community com2008 members target:65001:3 set routing-instances vlan2007 routing-options auto-export set routing-instances vlan2007 instance-type vrf set routing-instances vlan2007 interface irb.2007 set routing-instances vlan2007 route-distinguisher 10.179.1.1:2 set routing-instances vlan2007 vrf-import EVPN-IMPORT set routing-instances vlan2007 vrf-target target:65001:2 set routing-instances vlan2008 routing-options auto-export set routing-instances vlan2008 instance-type vrf set routing-instances vlan2008 interface irb.2008 set routing-instances vlan2008 route-distinguisher 10.179.1.1:3 set routing-instances vlan2008 vrf-import EVPN-IMPORT2008 set routing-instances vlan2008 vrf-target target:65001:3 set routing-options static route 0.0.0.0/0 next-hop 10.193.0.1 set routing-options static route 0.0.0.0/0 no-readvertise set routing-options static route 10.179.1.0/24 discard set routing-options forwarding-table export PFE-ECMP set routing-options forwarding-table ecmp-fast-reroute set routing-options router-id 10.179.1.1 set routing-options autonomous-system 64601 set protocols evpn vni-options vni 102005 vrf-target target:1:2005 set protocols evpn vni-options vni 102007 vrf-target target:1:2007 set protocols evpn vni-options vni 102008 vrf-target target:1:2008 set protocols evpn encapsulation vxlan set protocols evpn multicast-mode ingress-replication set protocols evpn default-gateway no-gateway-community set protocols evpn extended-vni-list all set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY mtu-discovery set protocols bgp group UNDERLAY export send-direct set protocols bgp group UNDERLAY local-as 65034 set protocols bgp group UNDERLAY multipath multiple-as set protocols bgp group UNDERLAY bfd-liveness-detection minimum-interval 1000 set protocols bgp group UNDERLAY bfd-liveness-detection multiplier 3 set protocols bgp group UNDERLAY bfd-liveness-detection session-mode automatic set protocols bgp group UNDERLAY neighbor 172.16.1.34 description leaf2 set protocols bgp group UNDERLAY neighbor 172.16.1.34 peer-as 65037 set protocols bgp group UNDERLAY neighbor 172.16.1.42 description leaf1 set protocols bgp group UNDERLAY neighbor 172.16.1.42 peer-as 65036 set protocols bgp group EVPN-IBGP type internal set protocols bgp group EVPN-IBGP local-address 10.179.1.1 set protocols bgp group EVPN-IBGP import import-overlay set protocols bgp group EVPN-IBGP family evpn signaling set protocols bgp group EVPN-IBGP cluster 0.0.0.1 set protocols bgp group EVPN-IBGP multipath set protocols bgp group EVPN-IBGP bfd-liveness-detection minimum-interval 350 set protocols bgp group EVPN-IBGP bfd-liveness-detection multiplier 3 set protocols bgp group EVPN-IBGP bfd-liveness-detection session-mode automatic set protocols bgp group EVPN-IBGP neighbor 10.179.1.5 description lfsw01 set protocols bgp group EVPN-IBGP neighbor 10.179.1.6 description lfsw02 set protocols bgp group EVPN-IBGP neighbor 10.179.1.2 description spine2 set protocols bgp group EVPN-IBGP neighbor 10.179.0.1 description dc1_spine1 set protocols bgp group EVPN-IBGP neighbor 10.179.0.2 description dc1_spine2 set protocols bgp group core type external set protocols bgp group core export export-dc-routes set protocols bgp group core peer-as 100 set protocols bgp group core local-as 65034 set protocols bgp group core neighbor 172.16.5.2 set protocols igmp-snooping vlan default set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 10.179.1.1:1 set switch-options vrf-target target:65512:1111 set vlans default vlan-id 1 set vlans private1-a-dal vlan-id 2005 set vlans private1-a-dal l3-interface irb.3502 set vlans private1-a-dal vxlan vni 102005 set vlans private1-a-sfo vlan-id 2007 set vlans private1-a-sfo l3-interface irb.2007 set vlans private1-a-sfo vxlan vni 102007 set vlans private1-b-sfo vlan-id 2008 set vlans private1-b-sfo l3-interface irb.2008 set vlans private1-b-sfo vxlan vni 102008
Spine2 configuration
set system host-name qfxssw30-dfw set interfaces xe-0/0/0 description link_leaf1 set interfaces xe-0/0/0 unit 0 family inet address 172.16.1.45/30 set interfaces xe-0/0/1 description link_leaf2 set interfaces xe-0/0/1 unit 0 family inet address 172.16.1.37/30 set interfaces xe-0/0/2 description link_cr2 set interfaces xe-0/0/2 unit 0 family inet address 172.16.5.5/30 set interfaces em0 unit 0 family inet address 10.193.0.83/24 set interfaces em1 unit 0 family inet address 169.254.0.2/24 set interfaces irb unit 2007 proxy-macip-advertisement set interfaces irb unit 2007 virtual-gateway-accept-data set interfaces irb unit 2007 family inet address 10.192.96.3/22 virtual-gateway-address 10.192.96.1 set interfaces irb unit 2008 proxy-macip-advertisement set interfaces irb unit 2008 virtual-gateway-accept-data set interfaces irb unit 2008 family inet address 10.192.112.3/22 virtual-gateway-address 10.192.112.1 set interfaces irb unit 3502 proxy-macip-advertisement set interfaces irb unit 3502 virtual-gateway-accept-data set interfaces irb unit 3502 family inet address 10.192.64.5/22 virtual-gateway-address 10.192.64.1 set interfaces lo0 unit 0 family inet address 10.179.1.2/32 set policy-options prefix-list fab-lo0s 10.179.1.0/24 set policy-options policy-statement EVPN-IMPORT term VNI102007 from community com2008 set policy-options policy-statement EVPN-IMPORT term VNI102007 then accept set policy-options policy-statement EVPN-IMPORT term VNI102005 from community com2005 set policy-options policy-statement EVPN-IMPORT term VNI102005 then accept set policy-options policy-statement EVPN-IMPORT2008 term 1 from community com2007 set policy-options policy-statement EVPN-IMPORT2008 term 1 then accept set policy-options policy-statement PFE-ECMP then load-balance per-packet set policy-options policy-statement export-dc-routes term fabric from protocol static set policy-options policy-statement export-dc-routes term fabric from prefix-list-filter fab-lo0s exact set policy-options policy-statement export-dc-routes term fabric then accept set policy-options policy-statement export-dc-routes then reject set policy-options policy-statement import-overlay term 1 from family evpn set policy-options policy-statement import-overlay term 1 from next-hop 10.179.0.1 set policy-options policy-statement import-overlay term 1 from nlri-route-type 1 set policy-options policy-statement import-overlay term 1 from nlri-route-type 2 set policy-options policy-statement import-overlay term 1 then reject set policy-options policy-statement import-overlay term 2 from family evpn set policy-options policy-statement import-overlay term 2 from next-hop 10.179.0.2 set policy-options policy-statement import-overlay term 2 from nlri-route-type 1 set policy-options policy-statement import-overlay term 2 from nlri-route-type 2 set policy-options policy-statement import-overlay term 2 then reject set policy-options policy-statement import-overlay then accept set policy-options policy-statement send-direct term send-lo0 from family inet set policy-options policy-statement send-direct term send-lo0 from protocol direct set policy-options policy-statement send-direct term send-lo0 then accept set policy-options community com2005 members target:65001:4 set policy-options community com2007 members target:65001:2 set policy-options community com2008 members target:65001:3 set routing-instances vlan2007 routing-options auto-export set routing-instances vlan2007 instance-type vrf set routing-instances vlan2007 interface irb.2007 set routing-instances vlan2007 route-distinguisher 10.179.1.1:2 set routing-instances vlan2007 vrf-import EVPN-IMPORT set routing-instances vlan2007 vrf-target target:65001:2 set routing-instances vlan2008 routing-options auto-export set routing-instances vlan2008 instance-type vrf set routing-instances vlan2008 interface irb.2008 set routing-instances vlan2008 route-distinguisher 10.179.1.1:3 set routing-instances vlan2008 vrf-import EVPN-IMPORT2008 set routing-instances vlan2008 vrf-target target:65001:3 set routing-options static route 0.0.0.0/0 next-hop 10.193.0.1 set routing-options static route 0.0.0.0/0 no-readvertise set routing-options static route 10.179.1.0/24 discard set routing-options forwarding-table export PFE-ECMP set routing-options forwarding-table ecmp-fast-reroute set routing-options router-id 10.179.1.2 set routing-options autonomous-system 64601 set protocols evpn vni-options vni 102005 vrf-target target:1:2005 set protocols evpn vni-options vni 102007 vrf-target target:1:2007 set protocols evpn vni-options vni 102008 vrf-target target:1:2008 set protocols evpn encapsulation vxlan set protocols evpn multicast-mode ingress-replication set protocols evpn default-gateway no-gateway-community set protocols evpn extended-vni-list all set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY mtu-discovery set protocols bgp group UNDERLAY export send-direct set protocols bgp group UNDERLAY local-as 65035 set protocols bgp group UNDERLAY multipath multiple-as set protocols bgp group UNDERLAY bfd-liveness-detection minimum-interval 1000 set protocols bgp group UNDERLAY bfd-liveness-detection multiplier 3 set protocols bgp group UNDERLAY bfd-liveness-detection session-mode automatic set protocols bgp group UNDERLAY neighbor 172.16.1.38 description leaf2 set protocols bgp group UNDERLAY neighbor 172.16.1.38 peer-as 65037 set protocols bgp group UNDERLAY neighbor 172.16.1.46 description leaf1 set protocols bgp group UNDERLAY neighbor 172.16.1.46 peer-as 65036 set protocols bgp group EVPN-IBGP type internal set protocols bgp group EVPN-IBGP local-address 10.179.1.2 set protocols bgp group EVPN-IBGP import import-overlay set protocols bgp group EVPN-IBGP family evpn signaling set protocols bgp group EVPN-IBGP cluster 0.0.0.2 set protocols bgp group EVPN-IBGP multipath set protocols bgp group EVPN-IBGP bfd-liveness-detection minimum-interval 350 set protocols bgp group EVPN-IBGP bfd-liveness-detection multiplier 3 set protocols bgp group EVPN-IBGP bfd-liveness-detection session-mode automatic set protocols bgp group EVPN-IBGP neighbor 10.179.1.5 description lfsw01 set protocols bgp group EVPN-IBGP neighbor 10.179.1.6 description lfsw02 set protocols bgp group EVPN-IBGP neighbor 10.179.1.1 description spine1 set protocols bgp group EVPN-IBGP neighbor 10.179.0.1 description dc1_spine1 set protocols bgp group EVPN-IBGP neighbor 10.179.0.2 description dc1_spine2 set protocols bgp group core type external set protocols bgp group core export export-dc-routes set protocols bgp group core peer-as 100 set protocols bgp group core local-as 65035 set protocols bgp group core neighbor 172.16.5.6 description cr2 set protocols igmp-snooping vlan default set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 10.179.1.2:1 set switch-options vrf-target target:65512:1111 set vlans default vlan-id 1 set vlans private1-a-dal vlan-id 2005 set vlans private1-a-dal l3-interface irb.3502 set vlans private1-a-dal vxlan vni 102005 set vlans private1-a-sfo vlan-id 2007 set vlans private1-a-sfo l3-interface irb.2007 set vlans private1-a-sfo vxlan vni 102007 set vlans private1-b-sfo vlan-id 2008 set vlans private1-b-sfo l3-interface irb.2008 set vlans private1-b-sfo vxlan vni 102008
leaf1 configuration
set system host-name qfxlsw29-dfw set interfaces xe-0/0/0 description link_spine1 set interfaces xe-0/0/0 unit 0 family inet address 172.16.1.42/30 set interfaces xe-0/0/1 description link_spine2 set interfaces xe-0/0/1 unit 0 family inet address 172.16.1.46/30 set interfaces xe-0/0/8 description srv205 set interfaces xe-0/0/8 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/8 unit 0 family ethernet-switching vlan members private1-a-dal set interfaces xe-0/0/9 description srv206 set interfaces xe-0/0/9 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/9 unit 0 family ethernet-switching vlan members private1-a-dal set interfaces xe-0/0/10 description srv20 set interfaces xe-0/0/10 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/10 unit 0 family ethernet-switching vlan members private1-b-sfo set interfaces xe-0/0/11 description srv10 set interfaces xe-0/0/11 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/11 unit 0 family ethernet-switching vlan members private1-a-sfo set interfaces em0 unit 0 family inet address 10.193.0.84/24 set interfaces lo0 unit 0 family inet address 10.179.1.5/32 set policy-options policy-statement PFE-ECMP then load-balance per-packet set policy-options policy-statement send-direct term send-lo0 from protocol direct set policy-options policy-statement send-direct term send-lo0 then accept set routing-options static route 0.0.0.0/0 next-hop 10.193.0.1 set routing-options static route 0.0.0.0/0 no-readvertise set routing-options forwarding-table export PFE-ECMP set routing-options forwarding-table ecmp-fast-reroute set routing-options router-id 10.179.1.5 set routing-options autonomous-system 64601 set protocols evpn vni-options vni 102005 vrf-target target:1:2005 set protocols evpn vni-options vni 102007 vrf-target target:1:2007 set protocols evpn vni-options vni 102008 vrf-target target:1:2008 set protocols evpn encapsulation vxlan set protocols evpn multicast-mode ingress-replication set protocols evpn extended-vni-list all set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY export send-direct set protocols bgp group UNDERLAY local-as 65036 set protocols bgp group UNDERLAY multipath multiple-as set protocols bgp group UNDERLAY bfd-liveness-detection minimum-interval 1000 set protocols bgp group UNDERLAY bfd-liveness-detection multiplier 3 set protocols bgp group UNDERLAY bfd-liveness-detection session-mode automatic set protocols bgp group UNDERLAY neighbor 172.16.1.41 description spine1 set protocols bgp group UNDERLAY neighbor 172.16.1.41 peer-as 65034 set protocols bgp group UNDERLAY neighbor 172.16.1.45 description spine2 set protocols bgp group UNDERLAY neighbor 172.16.1.45 peer-as 65035 set protocols bgp group EVPN-IBGP type internal set protocols bgp group EVPN-IBGP local-address 10.179.1.5 set protocols bgp group EVPN-IBGP family evpn signaling set protocols bgp group EVPN-IBGP multipath set protocols bgp group EVPN-IBGP bfd-liveness-detection minimum-interval 350 set protocols bgp group EVPN-IBGP bfd-liveness-detection multiplier 3 set protocols bgp group EVPN-IBGP bfd-liveness-detection session-mode automatic set protocols bgp group EVPN-IBGP neighbor 10.179.1.1 description spsw01 set protocols bgp group EVPN-IBGP neighbor 10.179.1.2 description spsw02 set protocols lldp disable set protocols lldp port-id-subtype interface-name set protocols lldp interface all set protocols lldp-med interface all set protocols igmp-snooping vlan default set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 10.179.1.5:1 set switch-options vrf-target target:65512:1111 set vlans default vlan-id 1 set vlans private1-a-dal vlan-id 2005 set vlans private1-a-dal vxlan vni 102005 set vlans private1-a-sfo vlan-id 2007 set vlans private1-a-sfo vxlan vni 102007 set vlans private1-b-sfo vlan-id 2008 set vlans private1-b-sfo vxlan vni 102008
leaf2 configuration
set system host-name qfxlsw30-dfw set interfaces xe-0/0/0 description link_spine1 set interfaces xe-0/0/0 unit 0 family inet address 172.16.1.34/30 set interfaces xe-0/0/1 description link_spine2 set interfaces xe-0/0/1 unit 0 family inet address 172.16.1.38/30 set interfaces xe-0/0/10 description srv42 set interfaces xe-0/0/10 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/10 unit 0 family ethernet-switching vlan members private1-b-sfo set interfaces xe-0/0/11 description srv41 set interfaces xe-0/0/11 unit 0 family ethernet-switching interface-mode access set interfaces xe-0/0/11 unit 0 family ethernet-switching vlan members private1-a-sfo set interfaces em0 unit 0 family inet address 10.193.0.85/24 set interfaces lo0 unit 0 family inet address 10.179.1.6/32 set forwarding-options storm-control-profiles default all set policy-options policy-statement PFE-ECMP then load-balance per-packet set policy-options policy-statement send-direct term send-lo0 from protocol direct set policy-options policy-statement send-direct term send-lo0 then accept set routing-options static route 0.0.0.0/0 next-hop 10.193.0.1 set routing-options static route 0.0.0.0/0 no-readvertise set routing-options forwarding-table export PFE-ECMP set routing-options forwarding-table ecmp-fast-reroute set routing-options router-id 10.179.1.6 set routing-options autonomous-system 64601 set protocols evpn vni-options vni 102007 vrf-target target:1:2007 set protocols evpn vni-options vni 102008 vrf-target target:1:2008 set protocols evpn encapsulation vxlan set protocols evpn multicast-mode ingress-replication set protocols evpn extended-vni-list all set protocols bgp group UNDERLAY type external set protocols bgp group UNDERLAY export send-direct set protocols bgp group UNDERLAY local-as 65037 set protocols bgp group UNDERLAY multipath multiple-as set protocols bgp group UNDERLAY bfd-liveness-detection minimum-interval 1000 set protocols bgp group UNDERLAY bfd-liveness-detection multiplier 3 set protocols bgp group UNDERLAY bfd-liveness-detection session-mode automatic set protocols bgp group UNDERLAY neighbor 172.16.1.33 description spine1 set protocols bgp group UNDERLAY neighbor 172.16.1.33 peer-as 65034 set protocols bgp group UNDERLAY neighbor 172.16.1.37 description spine2 set protocols bgp group UNDERLAY neighbor 172.16.1.37 peer-as 65035 set protocols bgp group EVPN-IBGP type internal set protocols bgp group EVPN-IBGP local-address 10.179.1.6 set protocols bgp group EVPN-IBGP family evpn signaling set protocols bgp group EVPN-IBGP multipath set protocols bgp group EVPN-IBGP bfd-liveness-detection minimum-interval 350 set protocols bgp group EVPN-IBGP bfd-liveness-detection multiplier 3 set protocols bgp group EVPN-IBGP bfd-liveness-detection session-mode automatic set protocols bgp group EVPN-IBGP neighbor 10.179.1.1 description spsw01 set protocols bgp group EVPN-IBGP neighbor 10.179.1.2 description spsw02 set protocols lldp disable set protocols lldp port-id-subtype interface-name set protocols lldp interface all set protocols lldp-med interface all set protocols igmp-snooping vlan default set switch-options service-id 1 set switch-options vtep-source-interface lo0.0 set switch-options route-distinguisher 10.179.1.6:1 set switch-options vrf-target target:65512:1111 set vlans default vlan-id 1 set vlans private1-a-sfo vlan-id 2007 set vlans private1-a-sfo vxlan vni 102007 set vlans private1-b-sfo vlan-id 2008 set vlans private1-b-sfo vxlan vni 102008
Site B EVPN/VXLAN testing & Verification
Underlay Verification
We are performing the test only on leaf1 and spine1, the same result will be also true for leaf2 and spine2. To verify this we will power off spine 1 later in this tutorial and make sure that spine2 will be taking all the traffic.
- on leaf1
root@qfxlsw29-dfw> show bgp summary group underlay
Threading mode: BGP I/O
Groups: 2 Peers: 4 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
100 50 0 0 0 0
inet.0
17 10 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn
State|#Active/Received/Accepted/Damped...
172.16.1.41 65034 42 42 0 33 15:47 Establ
inet.0: 3/6/6/0
172.16.1.45 65035 599 601 0 43 4:29:50 Establ
inet.0: 7/11/11/0
BGP sessions with spine1(1.41) ad spine2(1.45) are up.
root@qfxlsw29-dfw> show route advertising-protocol bgp 172.16.1.41 inet.0: 18 destinations, 27 routes (18 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.1.2/32 Self 65035 I * 10.179.1.5/32 Self I * 10.179.1.6/32 Self 65035 65037 I
root@qfxlsw29-dfw> show route advertising-protocol bgp 172.16.1.45 inet.0: 18 destinations, 27 routes (18 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.1.1/32 Self 65034 I * 10.179.1.5/32 Self I
root@qfxlsw29-dfw> show route receive-protocol bgp 172.16.1.41 table inet.0 inet.0: 18 destinations, 27 routes (18 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path *10.179.1.1/32 172.16.1.41 65034 I 10.179.1.2/32 172.16.1.41 65034 65037 65035 I 10.179.1.6/32 172.16.1.41 65034 65037 I
root@qfxlsw29-dfw> show route receive-protocol bgp 172.16.1.45 table inet.0 inet.0: 18 destinations, 27 routes (18 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path 10.179.1.1/32 172.16.1.45 65035 65037 65034 I * 10.179.1.2/32 172.16.1.45 65035 I * 10.179.1.6/32 172.16.1.45 65035 65037 I
-On spine1
root@qfxssw29-dfw> show bgp summary group underlay
Threading mode: BGP I/O
Groups: 3 Peers: 8 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
161 51 0 0 0 0
inet.0
26 9 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.16.1.34 65037 927 920 0 39 6:58:11 Establ
inet.0: 4/12/12/0
172.16.1.42 65036 233 231 0 42 1:43:11 Establ
inet.0: 4/12/12/0
root@qfxssw29-dfw> show route receive-protocol bgp 172.16.1.34 table inet.0 inet.0: 23 destinations, 42 routes (23 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.1.2/32 172.16.1.34 65037 65035 I 10.179.1.5/32 172.16.1.34 65037 65035 65036 I * 10.179.1.6/32 172.16.1.34 65037 I
root@qfxssw29-dfw> show route receive-protocol bgp 172.16.1.42 table inet.0 inet.0: 23 destinations, 42 routes (23 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path 10.179.1.2/32 172.16.1.42 65036 65035 I * 10.179.1.5/32 172.16.1.42 65036 I 10.179.1.6/32 172.16.1.42 65036 65035 65037 I
root@qfxssw29-dfw> show route advertising-protocol bgp 172.16.1.34 inet.0: 23 destinations, 42 routes (23 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.1.1/32 Self I * 10.179.1.5/32 Self 65036 I
root@qfxssw29-dfw> show route advertising-protocol bgp 172.16.1.42 inet.0: 23 destinations, 42 routes (23 active, 0 holddown, 0 hidden) Prefix Nexthop MED Lclpref AS path * 10.179.1.1/32 Self I * 10.179.1.2/32 Self 65037 65035 I * 10.179.1.6/32 Self 65037 I
Overlay Verification
- On leaf1
root@qfxlsw29-dfw> show bgp summary group EVPN-IBGP
Threading mode: BGP I/O
Groups: 2 Peers: 4 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
100 50 0 0 0 0
inet.0
17 10 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.179.1.1 64601 3319 3313 0 1 1d 1:04:31 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 32/50/50/0
default-switch.evpn.0: 32/50/50/0
10.179.1.2 64601 41509 41230 0 0 1w6d 0:15:29 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 18/50/50/0
default-switch.evpn.0: 18/50/50/0
- on spine1
root@qfxssw29-dfw> show bgp summary group EVPN-IBGP
Threading mode: BGP I/O
Groups: 3 Peers: 8 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
161 51 0 0 0 0
inet.0
26 9 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
10.179.1.2 64601 3316 3331 0 1 1d 1:05:54 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 18/51/51/0
default-switch.evpn.0: 18/43/43/0
10.179.1.5 64601 3317 3321 0 2 1d 1:05:49 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 11/11/11/0
default-switch.evpn.0: 11/11/11/0
10.179.1.6 64601 3315 3317 0 1 1d 1:05:36 Establ
__default_evpn__.evpn.0: 0/0/0/0
bgp.evpn.0: 6/6/6/0
default-switch.evpn.0: 6/6/6/0
Leaf1 can see spine1 and spine2 loopback and spine1 can see leaf1. leaf2 and spine2 loopback.
- Connectivity test
Test that srv10 with IP address 10.192.96.100 can ping 10.192.96.200 and 10.192.112.200 on leaf2.
Core network
cr1 configuration
set system host-name cr1 set system management-instance set chassis fpc 0 pic 0 interface-type xe set chassis fpc 0 performance-mode set interfaces xe-0/0/0 description link_spine1 set interfaces xe-0/0/0 unit 0 family inet address 172.16.3.2/30 set interfaces xe-0/0/1 description link_spine2 set interfaces xe-0/0/1 unit 0 family inet address 172.16.3.6/30 set interfaces xe-0/0/9 description link_p-router-xe-0/0/9 set interfaces xe-0/0/9 unit 0 family inet address 172.16.4.0/31 set interfaces xe-0/0/9 unit 0 family mpls set interfaces fxp0 unit 0 family inet address 10.193.0.86/24 set interfaces lo0 unit 0 family inet address 10.179.0.7/32 primary set snmp community junos1 routing-instance mgmt_junos set snmp routing-instance-access set routing-options static route 10.179.0.100/32 discard set routing-options autonomous-system 100 set protocols mpls no-cspf set protocols mpls interface xe-0/0/9.0 set protocols bgp group core_spine type external set protocols bgp group core_spine as-override set protocols bgp group core_spine neighbor 172.16.3.1 description spine1 set protocols bgp group core_spine neighbor 172.16.3.1 peer-as 65030 set protocols bgp group core_spine neighbor 172.16.3.5 description spine2 set protocols bgp group core_spine neighbor 172.16.3.5 peer-as 65031 set protocols bgp group core type internal set protocols bgp group core local-address 10.179.0.7 set protocols bgp group core export into-ibgp set protocols bgp group core neighbor 10.179.1.7 description cr2-lo0 set protocols ospf export into-ospf set protocols ospf area 0.0.0.0 interface lo0.0 passive set protocols ospf area 0.0.0.0 interface xe-0/0/9.0 interface-type p2p set protocols ldp egress-policy into-ldp set protocols ldp interface xe-0/0/9.0 set protocols ldp interface lo0.0 set policy-options prefix-list dc-routes 10.179.0.0/24 set policy-options policy-statement into-ibgp term 1 from protocol bgp set policy-options policy-statement into-ibgp term 1 from prefix-list-filter dc-routes exact set policy-options policy-statement into-ibgp term 1 then next-hop 10.179.0.100 set policy-options policy-statement into-ibgp term 1 then accept set policy-options policy-statement into-ibgp then reject set policy-options policy-statement into-ldp term 1 from interface lo0.0 set policy-options policy-statement into-ldp term 1 then accept set policy-options policy-statement into-ldp term 2 from protocol static set policy-options policy-statement into-ldp term 2 from route-filter 10.179.0.100/32 exact set policy-options policy-statement into-ldp term 2 then accept set policy-options policy-statement into-ldp then reject set policy-options policy-statement into-ospf term 1 from route-filter 10.179.0.100/32 exact set policy-options policy-statement into-ospf term 1 then accept set policy-options policy-statement into-ospf then reject set routing-instances mgmt_junos routing-options static route 0.0.0.0/0 next-hop 10.193.0.1
cr2 configuration
set system host-name cr2 set system management-instance set chassis fpc 0 pic 0 interface-type xe set chassis fpc 0 performance-mode set interfaces xe-0/0/0 description link_spine1 set interfaces xe-0/0/0 unit 0 family inet address 172.16.5.2/30 set interfaces xe-0/0/1 description link_spine2 set interfaces xe-0/0/1 unit 0 family inet address 172.16.5.6/30 set interfaces xe-0/0/9 description link_p-router-xe-0/0/8 set interfaces xe-0/0/9 unit 0 family inet address 172.16.6.0/31 set interfaces xe-0/0/9 unit 0 family mpls set interfaces fxp0 unit 0 family inet address 10.193.0.87/24 set interfaces lo0 unit 0 family inet address 10.179.1.7/32 primary set snmp community junos1 routing-instance mgmt_junos set snmp routing-instance-access set routing-options static route 10.179.1.100/32 discard set routing-options autonomous-system 100 set protocols mpls no-cspf set protocols mpls interface xe-0/0/9.0 set protocols bgp group core_spine type external set protocols bgp group core_spine peer-as 65034 set protocols bgp group core_spine as-override set protocols bgp group core_spine neighbor 172.16.5.1 description spine1 set protocols bgp group core type internal set protocols bgp group core local-address 10.179.1.7 set protocols bgp group core export into-ibgp set protocols bgp group core neighbor 10.179.0.7 description cr1-lo0 set protocols ospf export into-ospf set protocols ospf area 0.0.0.0 interface lo0.0 passive set protocols ospf area 0.0.0.0 interface xe-0/0/9.0 interface-type p2p set protocols ldp egress-policy into-ldp set protocols ldp interface xe-0/0/9.0 set protocols ldp interface lo0.0 set policy-options prefix-list dc-routes 10.179.1.0/24 set policy-options policy-statement into-ibgp term 1 from protocol bgp set policy-options policy-statement into-ibgp term 1 from prefix-list-filter dc-routes exact set policy-options policy-statement into-ibgp term 1 then next-hop 10.179.1.100 set policy-options policy-statement into-ibgp term 1 then accept set policy-options policy-statement into-ibgp then reject set policy-options policy-statement into-ldp term 1 from interface lo0.0 set policy-options policy-statement into-ldp term 1 then accept set policy-options policy-statement into-ldp term 2 from protocol static set policy-options policy-statement into-ldp term 2 from route-filter 10.179.1.100/32 exact set policy-options policy-statement into-ldp term 2 then accept set policy-options policy-statement into-ldp then reject set policy-options policy-statement into-ospf term 1 from route-filter 10.179.1.100/32 exact set policy-options policy-statement into-ospf term 1 then accept set policy-options policy-statement into-ospf then reject set routing-instances mgmt_junos routing-options static route 0.0.0.0/0 next-hop 10.193.0.1
p-router configuration
set system host-name p-router set system management-instance set chassis fpc 0 pic 0 interface-type xe set chassis fpc 0 performance-mode set interfaces xe-0/0/8 description link_cr2-dfw set interfaces xe-0/0/8 unit 0 family inet address 172.16.6.1/31 set interfaces xe-0/0/8 unit 0 family mpls set interfaces xe-0/0/9 description link_cr1-dfw set interfaces xe-0/0/9 unit 0 family inet address 172.16.4.1/31 set interfaces xe-0/0/9 unit 0 family mpls set interfaces fxp0 unit 0 family inet address 10.193.0.90/24 set interfaces lo0 unit 0 family inet address 4.4.4.4/32 set protocols mpls no-cspf set protocols mpls interface xe-0/0/9.0 set protocols mpls interface xe-0/0/8.0 set protocols ospf area 0.0.0.0 interface lo0.0 passive set protocols ospf area 0.0.0.0 interface xe-0/0/9.0 interface-type p2p set protocols ospf area 0.0.0.0 interface xe-0/0/8.0 interface-type p2p set protocols ldp interface xe-0/0/8.0 set protocols ldp interface xe-0/0/9.0 set protocols ldp interface lo0.0 set routing-instances mgmt_junos routing-options static route 0.0.0.0/0 next-hop 10.193.0.1 set routing-instances mgmt_junos routing-options static route 0.0.0.0/0 no-readvertise
Verification
- verify that OSPF sessions are up with site A router and site B router.
root@p-router> show ospf neighbor Address Interface State ID Pri Dead 172.16.6.0 xe-0/0/8.0 Full 10.179.1.7 128 31 172.16.4.0 xe-0/0/9.0 Full 10.179.0.7 128 33
- Verify that p router can reach both site A and site B routers loopback and anycast addresses.
root@p-router> show route protocol ospf
inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.179.0.7/32 *[OSPF/10] 1w6d 08:45:37, metric 1
> to 172.16.4.0 via xe-0/0/9.0
10.179.0.100/32 *[OSPF/150] 1w6d 06:48:35, metric 0, tag 0
> to 172.16.4.0 via xe-0/0/9.0
10.179.1.7/32 *[OSPF/10] 1w6d 07:03:40, metric 1
> to 172.16.6.0 via xe-0/0/8.0
10.179.1.100/32 *[OSPF/150] 1w6d 07:03:40, metric 0, tag 0
> to 172.16.6.0 via xe-0/0/8.0
224.0.0.5/32 *[OSPF/10] 1w6d 08:58:36, metric 1
MultiRecv
- Verify that ldp session is up
root@p-router> show ldp session Address State Connection Hold time Adv. Mode 10.179.0.7 Operational Open 26 DU 10.179.1.7 Operational Open 23 DU
root@p-router> show ldp database
Input label database, 4.4.4.4:0--10.179.0.7:0
Labels received: 3
Label Prefix
299776 4.4.4.4/32
3 10.179.0.7/32
3 10.179.0.100/32
299792 10.179.1.7/32
299792 10.179.1.100/32
Output label database, 4.4.4.4:0--10.179.0.7:0
Labels advertised: 3
Label Prefix
3 4.4.4.4/32
299776 10.179.0.7/32
299776 10.179.0.100/32
299792 10.179.1.7/32
299792 10.179.1.100/32
Input label database, 4.4.4.4:0--10.179.1.7:0
Labels received: 3
Label Prefix
299776 4.4.4.4/32
299792 10.179.0.7/32
299792 10.179.0.100/32
3 10.179.1.7/32
3 10.179.1.100/32
Output label database, 4.4.4.4:0--10.179.1.7:0
Labels advertised: 3
Label Prefix
3 4.4.4.4/32
299776 10.179.0.7/32
299776 10.179.0.100/32
299792 10.179.1.7/32
299792 10.179.1.100/32
- Last things to verify is the mpls table
root@p-router> show route table mpls.0
mpls.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
0 *[MPLS/0] 1w6d 09:03:53, metric 1
to table inet.0
0(S=0) *[MPLS/0] 1w6d 09:03:53, metric 1
to table mpls.0
1 *[MPLS/0] 1w6d 09:03:53, metric 1
Receive
2 *[MPLS/0] 1w6d 09:03:53, metric 1
to table inet6.0
2(S=0) *[MPLS/0] 1w6d 09:03:53, metric 1
to table mpls.0
13 *[MPLS/0] 1w6d 09:03:53, metric 1
Receive
299776 *[LDP/9] 1w6d 06:55:45, metric 1
> to 172.16.4.0 via xe-0/0/9.0, Pop
299776(S=0) *[LDP/9] 1w6d 06:55:45, metric 1
> to 172.16.4.0 via xe-0/0/9.0, Pop
299792 *[LDP/9] 1w6d 07:03:22, metric 1
> to 172.16.6.0 via xe-0/0/8.0, Pop
299792(S=0) *[LDP/9] 1w6d 07:03:22, metric 1
> to 172.16.6.0 via xe-0/0/8.0, Pop