vMX
Highlighted
vMX

need vMX evpn/vxlan L3 gateway assistance

[ Edited ]
4 weeks ago

I have the following topology. For easy troubleshooting, I turned off spine-7 and vtep-1.

evpn-L3.PNG

 

vServer-4 and PC on vtep-2 vlan 10 with vni 5010, vServer-5 on vtep-3 vlan 20 with vni 5020.

My issue is after I cleared bgp, L3 gateway works, but for a couple of hundreds of packets, then stopped. I ran packet capture on ge-0/0/1 on spine-6 to get the following arp requests with no replies

arp.PNG

the configuration are as follows:

 

root@spine-6# show interfaces irb | display set
set interfaces irb unit 10 proxy-macip-advertisement
set interfaces irb unit 10 virtual-gateway-accept-data
set interfaces irb unit 10 family inet address 10.1.1.101/24 virtual-gateway-address 10.1.1.100
set interfaces irb unit 20 proxy-macip-advertisement
set interfaces irb unit 20 virtual-gateway-accept-data
set interfaces irb unit 20 family inet address 10.1.2.101/24 virtual-gateway-address 10.1.2.100

[edit]
root@spine-6# show protocols | display set
set protocols bgp group underlay type external
set protocols bgp group underlay export exp2bgp
set protocols bgp group underlay local-as 65101
set protocols bgp group underlay multipath multiple-as
set protocols bgp group underlay neighbor 172.16.61.1 peer-as 65201
set protocols bgp group underlay neighbor 172.16.62.2 peer-as 65202
set protocols bgp group underlay neighbor 172.16.63.3 peer-as 65203
set protocols bgp group overlay type internal
set protocols bgp group overlay local-address 192.168.100.6
set protocols bgp group overlay family evpn signaling
set protocols bgp group overlay cluster 6.6.6.6
set protocols bgp group overlay multipath
set protocols bgp group overlay neighbor 192.168.100.1
set protocols bgp group overlay neighbor 192.168.100.2
set protocols bgp group overlay neighbor 192.168.100.3
set protocols bgp group overlay neighbor 192.168.100.7

[edit]
root@spine-6# show routing-instances | display set
set routing-instances inst-evpn vtep-source-interface lo0.0
set routing-instances inst-evpn instance-type virtual-switch
set routing-instances inst-evpn route-distinguisher 192.168.100.6:6
set routing-instances inst-evpn vrf-target target:64513:64513
set routing-instances inst-evpn protocols evpn encapsulation vxlan
set routing-instances inst-evpn protocols evpn extended-vni-list 5010
set routing-instances inst-evpn protocols evpn extended-vni-list 5020
set routing-instances inst-evpn protocols evpn default-gateway no-gateway-community
set routing-instances inst-evpn bridge-domains bd10 vlan-id 10
set routing-instances inst-evpn bridge-domains bd10 routing-interface irb.10
set routing-instances inst-evpn bridge-domains bd10 vxlan vni 5010
set routing-instances inst-evpn bridge-domains bd20 vlan-id 20
set routing-instances inst-evpn bridge-domains bd20 routing-interface irb.20
set routing-instances inst-evpn bridge-domains bd20 vxlan vni 5020
root@vtep-2# show protocols | display set
set protocols bgp group underlay type external
set protocols bgp group underlay export exp2bgp
set protocols bgp group underlay local-as 65202
set protocols bgp group underlay neighbor 172.16.72.7 peer-as 65102
set protocols bgp group underlay neighbor 172.16.62.6 peer-as 65101
set protocols bgp group overlay type internal
set protocols bgp group overlay local-address 192.168.100.2
set protocols bgp group overlay family evpn signaling
set protocols bgp group overlay neighbor 192.168.100.6
set protocols bgp group overlay neighbor 192.168.100.7
set protocols evpn encapsulation vxlan
set protocols evpn extended-vni-list 5010
set protocols evpn extended-vni-list 5020


root@vtep-2# show switch-options | display set
set switch-options vtep-source-interface lo0.0
set switch-options route-distinguisher 192.168.100.2:2
set switch-options vrf-target target:64513:64513

root@vtep-3# show protocols | display set
set protocols bgp group underlay type external
set protocols bgp group underlay export exp2bgp
set protocols bgp group underlay local-as 65203
set protocols bgp group underlay neighbor 172.16.73.7 peer-as 65102
set protocols bgp group underlay neighbor 172.16.63.6 peer-as 65101
set protocols bgp group overlay type internal
set protocols bgp group overlay local-address 192.168.100.3
set protocols bgp group overlay family evpn signaling
set protocols bgp group overlay neighbor 192.168.100.6
set protocols bgp group overlay neighbor 192.168.100.7
set protocols evpn encapsulation vxlan
set protocols evpn extended-vni-list 5010
set protocols evpn extended-vni-list 5020

root@vtep-3# show switch-options | display set
set switch-options vtep-source-interface lo0.0
set switch-options route-distinguisher 192.168.100.3:3
set switch-options vrf-target target:64513:64513


 

 

How can I troubleshoot this ?

 

thanks a lot !!

10 REPLIES 10
Highlighted
vMX

Re: need vMX evpn/vxlan L3 gateway assistance

4 weeks ago

 

Greetings, Please try the following:
 
 set interface irb unit <unit id> family inet address <address> preferred
  set interface irb unit <unit id> family inet6 address <address> preferred

If VGA IP is lower than IRB IP, ARPing would fail. IRB IP address is needed to process ARP requests and the default lower IP is preferred. So, if VGA has a lower IP than IRB, configure the “preferred” knob for IRB IP.

For more details, refer to the documentation on virtual-gateway-address.

 

 

I would also recommend you to enable the virtual-gateway-v4-mac knob. After configuring this knob when ping is done to IRB, then the relay packet would also start using configure VMAC as SMAC rather IRB MAC.
For more details, refer to the documentation on  virtual-gateway-v4-mac

 

 

If this solves your problem, please mark this post as "Accepted Solution" so we can help others too \:)/

Regards,

Lil Dexx
JNCIE-ENT#863, 3X JNCIP-[SP-ENT-DC], 4X JNCIA [cloud-DevOps-Junos-Design], Champions Ingenius, SSYB

Highlighted
vMX

Re: need vMX evpn/vxlan L3 gateway assistance

[ Edited ]
4 weeks ago

thanks a lot for taking a look and providing some suggestions.

I tried these, but the issue still occurs.

The symptom is that the traffic comes back after I reset bgp on spine-6. But after some time, the connection is broken.

I have the following packet captures from spine-6.

on interface ge-0/0/2, arp succeeds.

Int-02.PNG

but packet capture on int ge-0/0/1 shows arp fails.

Int-01.PNG

Any ideas ?

Also on spine-6, I have the following in bgp.evpn.0 when the connection is broken.

root@spine-6# ...e bgp.evpn.0 evpn-ethernet-tag-id 5020 | find 10.1.2.51
2:192.168.100.3:3::5020::00:05:86:71:8e:00::10.1.2.51/304 MAC/IP
                   *[BGP/170] 00:20:09, localpref 100, from 192.168.100.3
                      AS path: I, validation-state: unverified
                    > to 172.16.63.3 via ge-0/0/2.0
2:192.168.100.3:3::5020::aa:bb:cc:00:f0:00::10.1.2.52/304 MAC/IP
                   *[BGP/170] 00:20:09, localpref 100, from 192.168.100.3
                      AS path: I, validation-state: unverified
                    > to 172.16.63.3 via ge-0/0/2.0
2:192.168.100.6:6::5020::2c:6b:f5:8b:f7:f0::10.1.2.98/304 MAC/IP
                   *[EVPN/170] 00:20:08
                      Indirect
2:192.168.100.6:6::5020::2c:6b:f5:8b:f7:f0::10.1.2.100/304 MAC/IP
                   *[EVPN/170] 00:20:08
                      Indirect
3:192.168.100.2:2::5020::192.168.100.2/248 IM
                   *[BGP/170] 00:20:07, localpref 100, from 192.168.100.2
                      AS path: I, validation-state: unverified
                    > to 172.16.62.2 via ge-0/0/1.0
3:192.168.100.3:3::5020::192.168.100.3/248 IM
                   *[BGP/170] 00:20:09, localpref 100, from 192.168.100.3
                      AS path: I, validation-state: unverified
                    > to 172.16.63.3 via ge-0/0/2.0
3:192.168.100.6:6::5020::192.168.100.6/248 IM
                   *[EVPN/170] 00:20:06
                      Indirect

thanks again for help !!

Highlighted
vMX

Re: need vMX evpn/vxlan L3 gateway assistance

3 weeks ago

Hey 

 

Greetings again, the configuration looks good to me, might be a proxy-arp problem, can you try the no-arp-suppression knob, this  goes under the VLAN https://www.juniper.net/documentation/en_US/junos/topics/concept/evpn-proxy-arp-support.html

 

 

If this solves your problem, please mark this post as "Accepted Solution" so we can help others too \:)/

Regards,

Lil Dexx
JNCIE-ENT#863, 3X JNCIP-[SP-ENT-DC], 4X JNCIA [cloud-DevOps-Junos-Design], Champions Ingenius, SSYB

 

Highlighted
vMX

Re: need vMX evpn/vxlan L3 gateway assistance

3 weeks ago

Juniper web says:

The instance type 'virtual-switch' supports under hierarchy:

 

[edit logical-systems logical-system-name routing-instances routing-instance-name vlans vlan-name]
[edit routing-instances routing-instance-name vlans vlan-name]
[edit vlans vlan-name]
 I can't find this option there.
still looking .
 
thanks !!
 
Highlighted
vMX

Re: need vMX evpn/vxlan L3 gateway assistance

3 weeks ago

Hey gongyayu,

 

Try it under the bridge domain:

 

root@R1# show | compare
[edit routing-instances]
+ TEST {
+ instance-type virtual-switch;
+ bridge-domains {
+ TEST {
+ domain-type bridge;
+ vlan-id 1000;
+ no-arp-suppression;
+ routing-interface irb.1000;
+ }
+ }
+ }

 


If this solves your problem, please mark this post as "Accepted Solution" so we can help others too \:)/

Regards,

Lil Dexx
JNCIE-ENT#863, 3X JNCIP-[SP-ENT-DC], 4X JNCIA [cloud-DevOps-Junos-Design], Champions Ingenius, SSYB

Highlighted
vMX

Re: need vMX evpn/vxlan L3 gateway assistance

3 weeks ago

thanks so much !

I did this, but not fixed my issue.

vXLAN.PNG

After I cleared BGP, the connectivity resumed, but it worked for a couple of packets, then stopped.

Do you think this might be cause vMX?

I am deploying DCI via EVPN/vXLAN, right now not heavily use L3 Gateway yet.  L2 gateway works fine.

Also I  am taking ADCX training and work the training Lab.

 

thanks so much for your help !!

Highlighted
vMX

Re: need vMX evpn/vxlan L3 gateway assistance

3 weeks ago

sorry for a typo.

a couple of hundreds of packets. then stopped.

Note: without no-arp-suppression. with this, clear bgp neighbor does not help at all.

Highlighted
vMX

Re: need vMX evpn/vxlan L3 gateway assistance

3 weeks ago

Hey 

 

Here is an IP fabric with EVPN/VXLAN already up and running https://jlabs.juniper.net/vlabs/portal/ip-fabric-evpn-vxlan/index.page

 

If this solves your problem, please mark this post as "Accepted Solution" so we can help others too \:)/

 

Regards,

 

Lil Dexx 
JNCIE-ENT#863, 3X JNCIP-[SP-ENT-DC], 4X JNCIA [cloud-DevOps-Junos-Design], Champions Ingenius, SSYB

Highlighted
vMX

Re: need vMX evpn/vxlan L3 gateway assistance

3 weeks ago

thanks so much.

I tried L3 VGA on vQFX. it works fine.

Highlighted
vMX

Re: need vMX evpn/vxlan L3 gateway assistance

3 weeks ago

Hey 

 

 

If this solves your problem, please mark this post as "Accepted Solution" so we can help others too \:)/

Regards,

Lil Dexx
JNCIE-ENT#863, 3X JNCIP-[SP-ENT-DC], 4X JNCIA [cloud-DevOps-Junos-Design], Champions Ingenius, SSYB

Feedback