Segment Routing IPv6 (SRv6) with FRR and Ubuntu

27 Jun 2022 2328 words 11 minutes

Contents

It is no secret that Segment Routing over MPLS offers a lot of promise and provides a simple path for network operators to migrate from existing LDP and RSVP-TE based networks. However, what if I told you that you could do even more with SR and not even run MPLS at all? What if then I told you that these nodes could be located anywhere with IPv6 access and physical adjacency is not even required? Further down the spiral, what if end-machines could easily participate in Segment Routing themselves?

Let’s take the ‘red pill’ and talk about Segment Routing over IPv6 (SRv6). My personal favorite in terms of potential. Fair warning though, it currently can be rough on the bleeding edge but we take a look at one of my recent experiments with FRRouting and SRv6 L3VPNs on Ubuntu 20.04 LTS.

The Segment Routing Header (SRH)

With SR-MPLS, regular old MPLS labels are used to define behaviors about traffic. In the SRv6 world, we do not use MPLS at all. So how do we mark traffic in a similar way so that instructions can be encoded? Thanks to the elasticity of IPv6, a new header was created to solve this issue: the IPv6 Segment Routing Header (SRH). From a very high-level, it basically contains a list of IPv6 addresses to encode instructions that a transit or receiving router would understand the meaning of. The beauty of this is that because these are IPv6 addresses, as long as the network knows how to reach the next hop, it does not need to understand the meaning of the address if it is not participating in Segment Routing.

Unlike MPLS, where each node along the path needs to understand and perform an action on a label, this traffic appears to non-participating routers like any other IPv6 packet along a path to a destination. This is huge! This means that we can stitch up services we used to rely on MPLS for (like L3VPNs) between any two points anywhere that can reach each other over IPv6. Goodbye Inter-AS nightmares, anyone? Yes, please! And this only scratches the surface of future SDN possibilities.

Linux Support

For today’s post, this is where things start to get a little muddy. There are several endpoint functions in various states of support and whether it actually functions in my experience.

I was attempting to demonstrate IPv6 and IPv4 L3VPNs working over an SRv6 backbone. While IPv6 L3VPN mostly works (with a quirky caveat), I could not get IPv4 L3VPN to function at all on the following kernels:

5.10.0-10 (stock Debian 11)
5.18.6 (custom compiled)
5.18.7 (custom compiled)

All with the latest FRR version at the time of this writing: 8.2.2

Prior this patch adding support for the vrftable argument for iproute2, you will likely be unable to even attempt to install IPv4 End.DT4 routes for IPv4 L3VPNs.

The following proc settings were used in addition to the FRR recommendations:

net.vrf.strict_mode = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv6.seg6_flowlabel = 1
net.ipv6.conf.all.seg6_enabled = 1

FRR Configuration

The FRR configuration is quite simple, however pay attention to the SRv6 Locator. This must be a sufficient prefix size to allow for the generation of IPv6 addresses that become the instructions in the SRH. A /64 for each router serves this purpose quite well and I found it useful to summarize it with the loopback address used for BGP peering.

Attributes of the router:

Segment Locator Block: 2001:db8:1::/64 (name: UBNT1_SRV6)
Router Loopback: 2001:db8:1::/128
SRv6 Interfaces: ens3
- IS-IS Level 2 (49.0001.0000.0000.0001.00)
- This is merely to support discovery of the Loopback addresses and not directly SRv6 related
L3VPN VRF Interface: ens4 (name: VRF142)
- IPv6 address: 2001:db8:a::/64
- IPv4 address: 192.168.10.1/24

The configuration is identical on each router with interface attributes, BGP RD and IS-IS NET changed per router.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102


!
frr version 8.2.2
frr defaults traditional
hostname ubnt1
log syslog informational
service integrated-vtysh-config
!
interface ens3
 description SRv6
 ipv6 router isis SRv6
 isis circuit-type level-2-only
exit
!
interface ens4
 description VRF142
 ip address 192.168.10.1/24
 ipv6 address 2001:db8:a::/64
exit
!
interface lo
 description test
 ipv6 address 2001:db8:1::/128
 ipv6 router isis SRv6
 isis passive
exit
!
router bgp 65000
 bgp router-id 10.0.0.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 bgp graceful-restart
 neighbor 2001:db8:2:: remote-as 65000
 neighbor 2001:db8:2:: description ubnt2
 neighbor 2001:db8:2:: update-source lo
 neighbor 2001:db8:2:: capability extended-nexthop
 !
 segment-routing srv6
  locator UBNT1_SRV6
 exit
 !
 address-family ipv4 vpn
  neighbor 2001:db8:2:: activate
 exit-address-family
 !
 address-family ipv6 unicast
  network 2001:db8:1::/128
  aggregate-address 2001:db8:1::/64 summary-only
  neighbor 2001:db8:2:: activate
 exit-address-family
 !
 address-family ipv6 vpn
  neighbor 2001:db8:2:: activate
 exit-address-family
exit
!
router bgp 65000 vrf VRF142
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 bgp graceful-restart
 !
 address-family ipv4 unicast
  redistribute connected
  sid vpn export auto
  rd vpn export 65000:1
  nexthop vpn export 2001:db8:1::
  rt vpn both 65000:1
  export vpn
  import vpn
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
  sid vpn export auto
  rd vpn export 65000:1
  nexthop vpn export 2001:db8:1::
  rt vpn both 65000:1
  export vpn
  import vpn
 exit-address-family
exit
!
router isis SRv6
 is-type level-2-only
 net 49.0001.0000.0000.0001.00
 lsp-mtu 1300
 topology ipv6-unicast
exit
!
segment-routing
 srv6
  locators
   locator UBNT_SRV6
    prefix 2001:db8:1::/64
   exit
   !
  exit
  !
 exit
 !
exit
!
end

FRR Verification

Within FRR (vtysh), we can verify several attributes about the routing and SRv6 status:

Verify the global IPv6 routing table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


ubnt1# show ipv6 route
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* ::/0 [0/100] via fe80::5201:ff:fe05:0, ens3, weight 1, 01:57:15
B>* 2001:db8:1::/64 [200/0] unreachable (blackhole), weight 1, 03:04:45
C>* 2001:db8:1::/128 is directly connected, lo, 03:04:45
B>r 2001:db8:1:0:100::/128 [20/0] is directly connected, VRF142, seg6local End.DT4 table 142, seg6 ::, weight 1, 03:04:45
B>* 2001:db8:1:0:200::/128 [20/0] is directly connected, VRF142, seg6local End.DT6 table 142, seg6 ::, weight 1, 03:04:45
B>  2001:db8:2::/64 [200/0] via 2001:db8:2:: (recursive), weight 1, 03:04:45
  *                           via fe80::c4c9:b0ff:fea5:a0b1, ens3, weight 1, 03:04:45
I>* 2001:db8:2::/128 [115/20] via fe80::c4c9:b0ff:fea5:a0b1, ens3, weight 1, 03:04:45
C * fe80::/64 is directly connected, ens3, 03:04:45
C>* fe80::/64 is directly connected, ens4, 03:04:45

If you noticed the rejected route on the End.DT4 segment, we get to that later.

Verify the status of the SRv6 locator:

1
2
3
4
5


ubnt1# show segment-routing srv6 locator 
Locator:
Name                 ID      Prefix                   Status
-------------------- ------- ------------------------ -------
UBNT1_SRV6                    1 2001:db8:1::/64          Up

Verify the status of BGP’s use of the locator:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


ubnt1# show bgp segment-routing srv6 
locator_name: UBNT1_SRV6
locator_chunks:
- 2001:db8:1::/64
functions:
- sid: 2001:db8:1:0:100::
  locator: UBNT1_SRV6
- sid: 2001:db8:1:0:200::
  locator: UBNT1_SRV6
bgps:
- name: default
  vpn_policy[AFI_IP].tovpn_sid: none
  vpn_policy[AFI_IP6].tovpn_sid: none
- name: VRF142
  vpn_policy[AFI_IP].tovpn_sid: 2001:db8:1:0:100::
  vpn_policy[AFI_IP6].tovpn_sid: 2001:db8:1:0:200::

Verify the IPv4 VRF142 routing table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


ubnt1# show ip route vrf VRF142
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF VRF142:
C>* 192.168.10.0/24 is directly connected, ens4, 00:09:35
B>  192.168.20.0/24 [20/0] via 2001:db8:2:: (vrf default) (recursive), label 4096, seg6local unspec unknown(seg6local_context2str), seg6 2001:db8:2:0:100::, weight 1, 00:09:32
  *                          via fe80::c4c9:b0ff:fea5:a0b1, ens3 (vrf default), label 4096, seg6local unspec unknown(seg6local_context2str), seg6 2001:db8:2:0:100::, weight 1, 00:09:32

Verify the IPv6 VRF142 routing table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


ubnt1# show ipv6 route vrf VRF142
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF VRF142:
C>* 2001:db8:a::/64 is directly connected, ens4, 00:10:45
B>  2001:db8:b::/64 [20/0] via 2001:db8:2:: (vrf default) (recursive), label 8192, seg6local unspec unknown(seg6local_context2str), seg6 2001:db8:2:0:200::, weight 1, 00:10:44
  *                           via fe80::c4c9:b0ff:fea5:a0b1, ens3 (vrf default), label 8192, seg6local unspec unknown(seg6local_context2str), seg6 2001:db8:2:0:200::, weight 1, 00:10:44
C>* fe80::/64 is directly connected, ens4, 00:16:35

IPv6 VPN Testing

Under no circumstance was I able to make the remote router respond across the SRv6 network to its own interfaces in the VRF (for testing from router to router). For example (ubnt1 ens4 to ubnt2 ens4):

1
2
3
4
5


root@ubnt1:/home/jvoss# ip vrf exec VRF142 ping -I 2001:db8:a:: 2001:db8:b::
PING 2001:db8:b::(2001:db8:b::) from 2001:db8:a:: : 56 data bytes
^C
--- 2001:db8:b:: ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6151ms

These same test succeed and respond to icmp-echo when using other mechanisms such as MPLS.

Testing from a machine attached to the router inside the VRF (ens4) also fails to elicit a response from the remote routers VRF ens4 interface:

1
2
3
4
5


PC1#ping 2001:db8:b::
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:B::, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

However testing a PC on the other side of each router’s ens4 (VRF interface) works as expected:

1
2
3
4
5


PC1#ping 2001:db8:b::2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:B::2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/10 ms

IPv4 L3VPN Issues

NOTE: Here we can see the rejected route for the local IPv4 L3VPN by Zebra. This is preventing the IPv4 portion of this lab from working.

Attempting to install the route manually, however IPv4 traffic still fails to forward to the VRF:

1

ip route add 2001:db8:1:0:100::/128 encap seg6local action End.DT4 vrftable 142 dev VRF142

Testing from behind ubnt1 to ubnt2’s VRF142 interface:

1
2
3
4
5


PC1#ping 192.168.20.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.20.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Testing from behind ubnt1 to PC2 behind ubnt2’s VRF142 interface:

1
2
3
4
5


PC1#ping 192.168.20.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.20.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Traffic does appear to be encapsulated correctly to the destination so it seems to be an issue with the receiving router decapsulating the instruction. The following was observed on ubnt2 receiving pings from PC1 via ubnt1:

1
2
3
4
5
6
7
8


root@ubnt2:~# tcpdump -i ens3 host 2001:db8:2:0:100::
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
17:51:41.342725 IP6 fd9b:ee89:41b5:edcc:c799:9317:acda:ebf1 > 2001:db8:2:0:100::: srcrt (len=2, type=4, segleft=0[|srcrt]
17:51:43.340045 IP6 fd9b:ee89:41b5:edcc:c799:9317:acda:ebf1 > 2001:db8:2:0:100::: srcrt (len=2, type=4, segleft=0[|srcrt]
17:51:45.341147 IP6 fd9b:ee89:41b5:edcc:c799:9317:acda:ebf1 > 2001:db8:2:0:100::: srcrt (len=2, type=4, segleft=0[|srcrt]
17:51:47.341180 IP6 fd9b:ee89:41b5:edcc:c799:9317:acda:ebf1 > 2001:db8:2:0:100::: srcrt (len=2, type=4, segleft=0[|srcrt]
17:51:49.342127 IP6 fd9b:ee89:41b5:edcc:c799:9317:acda:ebf1 > 2001:db8:2:0:100::: srcrt (len=2, type=4, segleft=0[|srcrt]

No echo-request is seen on ubnt2 exiting the VRF142 interface (ens4) towards PC2.

FRR SRv6 End.DT4 troubleshooting

Examining the Zebra log revealed the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


2022/06/27 19:16:59 ZEBRA: [G08K1-416PW] netlink_nexthop_msg_encode: unsupport seg6local behaviour action=8
2022/06/27 19:16:59 ZEBRA: [G08K1-416PW] netlink_nexthop_msg_encode: unsupport seg6local behaviour action=8
2022/06/27 19:16:59 ZEBRA: [NZNZ4-7P54Y] default(0:254):2001:db8:1:0💯:/128: Processing rn 0x55747750b5d0
2022/06/27 19:16:59 ZEBRA: [ZJVZ4-XEGPF] default(0:254):2001:db8:1:0💯:/128: Examine re 0x55747750b050 (bgp) status: Changed flags: Recursion dist 20 metric 0
2022/06/27 19:16:59 ZEBRA: [JPJF4-TGCY5] default(0:254):2001:db8:1:0💯:/128: After processing: old_selected 0x0 new_selected 0x55747750b050 old_fib 0x0 new_fib 0x55747750b050
2022/06/27 19:16:59 ZEBRA: [ZFRN1-EZNJZ] default(0:254):2001:db8:1:0💯:/128: Adding route rn 0x55747750b5d0, re 0x55747750b050 (bgp)
2022/06/27 19:16:59 ZEBRA: [QZ1V6-CRT8D] default(0:254):2001:db8:1:0💯:/128 rn 0x55747750b5d0 dequeued from sub-queue 6
2022/06/27 19:16:59 ZEBRA: [NZNZ4-7P54Y] default(0:254):2001:db8:1:0:200::/128: Processing rn 0x55747750b950
2022/06/27 19:16:59 ZEBRA: [ZJVZ4-XEGPF] default(0:254):2001:db8:1:0:200::/128: Examine re 0x55747750abc0 (bgp) status: Changed flags: Recursion dist 20 metric 0
2022/06/27 19:16:59 ZEBRA: [JPJF4-TGCY5] default(0:254):2001:db8:1:0:200::/128: After processing: old_selected 0x0 new_selected 0x55747750abc0 old_fib 0x0 new_fib 0x55747750abc0
2022/06/27 19:16:59 ZEBRA: [ZFRN1-EZNJZ] default(0:254):2001:db8:1:0:200::/128: Adding route rn 0x55747750b950, re 0x55747750abc0 (bgp)
2022/06/27 19:16:59 ZEBRA: [QZ1V6-CRT8D] default(0:254):2001:db8:1:0:200::/128 rn 0x55747750b950 dequeued from sub-queue 6
2022/06/27 19:16:59 ZEBRA: [P2XBZ-RAFQ5][EC 4043309074] Failed to install Nexthop ID (76) into the kernel
2022/06/27 19:16:59 ZEBRA: [HSYZM-HV7HF] Extended Error: Nexthop id does not exist
2022/06/27 19:16:59 ZEBRA: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=32, pid=3300413719
2022/06/27 19:16:59 ZEBRA: [VYKYC-709DP] default(0:254):2001:db8:1:0💯:/128: Route install failed

Summary

SRv6 brings a lot of new potential and simplification of existing services that are offered via MPLS or derivatives. Although there are stumbling blocks at present, at least in the Linux world, I believe there is a bright future for SRv6.

I hope to update this post in the future once issues with the End.DT4 endpoint function are resolved.