# Segment Routing IPv6 (SRv6) with FRR and Ubuntu It is no secret that [Segment Routing over MPLS](sr-mpls-basic) offers a lot of promise and provides a simple path for network operators to migrate from existing LDP and RSVP-TE based networks. However, what if I told you that you could do *even more* with SR and not even run MPLS at all? What if then I told you that these nodes could be located anywhere with IPv6 access and physical adjacency is not even required? Further down the spiral, what if end-machines could easily participate in Segment Routing themselves? Let's take the 'red pill' and talk about Segment Routing over IPv6 (SRv6). My personal favorite in terms of potential. Fair warning though, it currently can be rough on the bleeding edge but we take a look at one of my recent experiments with [FRRouting](https://frrouting.org/) and SRv6 L3VPNs on Ubuntu 20.04 LTS. ## The Segment Routing Header (SRH) With SR-MPLS, regular old MPLS labels are used to define behaviors about traffic. In the SRv6 world, we do not use MPLS at all. So how do we mark traffic in a similar way so that instructions can be encoded? Thanks to the elasticity of IPv6, a new header was created to solve this issue: the [IPv6 Segment Routing Header (SRH)](https://datatracker.ietf.org/doc/html/draft-ietf-6man-segment-routing-header-15). From a very high-level, it basically contains a list of IPv6 addresses to encode instructions that a transit or receiving router would understand the meaning of. The beauty of this is that because these are IPv6 addresses, as long as the network knows how to reach the next hop, it does not need to understand the meaning of the address if it is not participating in Segment Routing. Unlike MPLS, where each node along the path needs to understand and perform an action on a label, this traffic appears to non-participating routers like any other IPv6 packet along a path to a destination. *This is huge!* This means that we can stitch up services we used to rely on MPLS for (like L3VPNs) between any two points *anywhere* that can reach each other over IPv6. Goodbye Inter-AS nightmares, anyone? Yes, please! And this only scratches the surface of future SDN possibilities. ## Linux Support For today's post, this is where things start to get a little muddy. There are several [endpoint functions](https://www.segment-routing.net/open-software/linux/) in various states of support and whether it actually functions in my experience. I was attempting to demonstrate IPv6 *and IPv4* L3VPNs working over an SRv6 backbone. While IPv6 L3VPN mostly works (with a quirky caveat), I could not get IPv4 L3VPN to function at all on the following kernels: * 5.10.0-10 (stock Debian 11) * 5.18.6 (custom compiled) * 5.18.7 (custom compiled) All with the latest FRR version at the time of this writing: 8.2.2 Prior [this patch](https://lkml.org/lkml/2020/11/23/1439) adding support for the `vrftable` argument for `iproute2`, you will likely be unable to even attempt to install IPv4 `End.DT4` routes for IPv4 L3VPNs. The following proc settings were used in addition to the [FRR recommendations](http://docs.frrouting.org/en/latest/installation.html): * net.vrf.strict_mode = 1 * net.ipv4.conf.all.rp_filter = 0 * net.ipv6.seg6_flowlabel = 1 * net.ipv6.conf.all.seg6_enabled = 1 ## FRR Configuration The FRR configuration is quite simple, however pay attention to the SRv6 Locator. This must be a sufficient prefix size to allow for the generation of IPv6 addresses that become the instructions in the SRH. A /64 for each router serves this purpose quite well and I found it useful to summarize it with the loopback address used for BGP peering. Attributes of the router: * Segment Locator Block: 2001:db8:1::/64 (name: UBNT1_SRV6) * Router Loopback: 2001:db8:1::/128 * SRv6 Interfaces: ens3 * IS-IS Level 2 (49.0001.0000.0000.0001.00) * *This is merely to support discovery of the Loopback addresses and not directly SRv6 related* * L3VPN VRF Interface: ens4 (name: VRF142) * IPv6 address: 2001:db8:a::/64 * IPv4 address: 192.168.10.1/24 The configuration is identical on each router with interface attributes, BGP RD and IS-IS NET changed per router. ```linux ! frr version 8.2.2 frr defaults traditional hostname ubnt1 log syslog informational service integrated-vtysh-config ! interface ens3 description SRv6 ipv6 router isis SRv6 isis circuit-type level-2-only exit ! interface ens4 description VRF142 ip address 192.168.10.1/24 ipv6 address 2001:db8:a::/64 exit ! interface lo description test ipv6 address 2001:db8:1::/128 ipv6 router isis SRv6 isis passive exit ! router bgp 65000 bgp router-id 10.0.0.1 bgp log-neighbor-changes no bgp default ipv4-unicast bgp graceful-restart neighbor 2001:db8:2:: remote-as 65000 neighbor 2001:db8:2:: description ubnt2 neighbor 2001:db8:2:: update-source lo neighbor 2001:db8:2:: capability extended-nexthop ! segment-routing srv6 locator UBNT1_SRV6 exit ! address-family ipv4 vpn neighbor 2001:db8:2:: activate exit-address-family ! address-family ipv6 unicast network 2001:db8:1::/128 aggregate-address 2001:db8:1::/64 summary-only neighbor 2001:db8:2:: activate exit-address-family ! address-family ipv6 vpn neighbor 2001:db8:2:: activate exit-address-family exit ! router bgp 65000 vrf VRF142 bgp log-neighbor-changes no bgp default ipv4-unicast bgp graceful-restart ! address-family ipv4 unicast redistribute connected sid vpn export auto rd vpn export 65000:1 nexthop vpn export 2001:db8:1:: rt vpn both 65000:1 export vpn import vpn exit-address-family ! address-family ipv6 unicast redistribute connected sid vpn export auto rd vpn export 65000:1 nexthop vpn export 2001:db8:1:: rt vpn both 65000:1 export vpn import vpn exit-address-family exit ! router isis SRv6 is-type level-2-only net 49.0001.0000.0000.0001.00 lsp-mtu 1300 topology ipv6-unicast exit ! segment-routing srv6 locators locator UBNT_SRV6 prefix 2001:db8:1::/64 exit ! exit ! exit ! exit ! end ``` ## FRR Verification Within FRR (`vtysh`), we can verify several attributes about the routing and SRv6 status: Verify the global IPv6 routing table: ```linux ubnt1# show ipv6 route Codes: K - kernel route, C - connected, S - static, R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure K>* ::/0 [0/100] via fe80::5201:ff:fe05:0, ens3, weight 1, 01:57:15 B>* 2001:db8:1::/64 [200/0] unreachable (blackhole), weight 1, 03:04:45 C>* 2001:db8:1::/128 is directly connected, lo, 03:04:45 B>r 2001:db8:1:0:100::/128 [20/0] is directly connected, VRF142, seg6local End.DT4 table 142, seg6 ::, weight 1, 03:04:45 B>* 2001:db8:1:0:200::/128 [20/0] is directly connected, VRF142, seg6local End.DT6 table 142, seg6 ::, weight 1, 03:04:45 B> 2001:db8:2::/64 [200/0] via 2001:db8:2:: (recursive), weight 1, 03:04:45 * via fe80::c4c9:b0ff:fea5:a0b1, ens3, weight 1, 03:04:45 I>* 2001:db8:2::/128 [115/20] via fe80::c4c9:b0ff:fea5:a0b1, ens3, weight 1, 03:04:45 C * fe80::/64 is directly connected, ens3, 03:04:45 C>* fe80::/64 is directly connected, ens4, 03:04:45 ``` If you noticed the rejected route on the `End.DT4` segment, we get to that [later](#ipv4-l3vpn-issues). Verify the status of the SRv6 locator: ```linux ubnt1# show segment-routing srv6 locator Locator: Name ID Prefix Status -------------------- ------- ------------------------ ------- UBNT1_SRV6 1 2001:db8:1::/64 Up ``` Verify the status of BGP's use of the locator: ```linux ubnt1# show bgp segment-routing srv6 locator_name: UBNT1_SRV6 locator_chunks: - 2001:db8:1::/64 functions: - sid: 2001:db8:1:0:100:: locator: UBNT1_SRV6 - sid: 2001:db8:1:0:200:: locator: UBNT1_SRV6 bgps: - name: default vpn_policy[AFI_IP].tovpn_sid: none vpn_policy[AFI_IP6].tovpn_sid: none - name: VRF142 vpn_policy[AFI_IP].tovpn_sid: 2001:db8:1:0:100:: vpn_policy[AFI_IP6].tovpn_sid: 2001:db8:1:0:200:: ``` Verify the IPv4 VRF142 routing table: ```linux ubnt1# show ip route vrf VRF142 Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF VRF142: C>* 192.168.10.0/24 is directly connected, ens4, 00:09:35 B> 192.168.20.0/24 [20/0] via 2001:db8:2:: (vrf default) (recursive), label 4096, seg6local unspec unknown(seg6local_context2str), seg6 2001:db8:2:0:100::, weight 1, 00:09:32 * via fe80::c4c9:b0ff:fea5:a0b1, ens3 (vrf default), label 4096, seg6local unspec unknown(seg6local_context2str), seg6 2001:db8:2:0:100::, weight 1, 00:09:32 ``` Verify the IPv6 VRF142 routing table: ```linux ubnt1# show ipv6 route vrf VRF142 Codes: K - kernel route, C - connected, S - static, R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF VRF142: C>* 2001:db8:a::/64 is directly connected, ens4, 00:10:45 B> 2001:db8:b::/64 [20/0] via 2001:db8:2:: (vrf default) (recursive), label 8192, seg6local unspec unknown(seg6local_context2str), seg6 2001:db8:2:0:200::, weight 1, 00:10:44 * via fe80::c4c9:b0ff:fea5:a0b1, ens3 (vrf default), label 8192, seg6local unspec unknown(seg6local_context2str), seg6 2001:db8:2:0:200::, weight 1, 00:10:44 C>* fe80::/64 is directly connected, ens4, 00:16:35 ``` ## IPv6 VPN Testing Under no circumstance was I able to make the remote router respond across the SRv6 network to its own interfaces in the VRF (for testing from router to router). For example (**ubnt1** `ens4` to **ubnt2** `ens4`): ```linux root@ubnt1:/home/jvoss# ip vrf exec VRF142 ping -I 2001:db8:a:: 2001:db8:b:: PING 2001:db8:b::(2001:db8:b::) from 2001:db8:a:: : 56 data bytes ^C --- 2001:db8:b:: ping statistics --- 7 packets transmitted, 0 received, 100% packet loss, time 6151ms ``` *These same test succeed and respond to icmp-echo when using other mechanisms such as MPLS.* Testing from a machine attached to the router inside the VRF (`ens4`) also fails to elicit a response from the remote routers VRF `ens4` interface: ```linux PC1#ping 2001:db8:b:: Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 2001:DB8:B::, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) ``` However testing a PC on the other side of each router's `ens4` (VRF interface) works as expected: ```linux PC1#ping 2001:db8:b::2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 2001:DB8:B::2, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/10 ms ``` ## IPv4 L3VPN Issues ![show ipv6 route](/posts/2022/06/srv6-frr/rDTR875uh5.png) NOTE: Here we can see the **rejected** route for the local IPv4 L3VPN by Zebra. This is preventing the IPv4 portion of this lab from working. Attempting to install the route manually, however IPv4 traffic still fails to forward to the VRF: ```bash ip route add 2001:db8:1:0:100::/128 encap seg6local action End.DT4 vrftable 142 dev VRF142 ``` Testing from behind **ubnt1** to **ubnt2**'s VRF142 interface: ```linux PC1#ping 192.168.20.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.20.1, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) ``` Testing from behind **ubnt1** to PC2 behind **ubnt2**'s VRF142 interface: ```linux PC1#ping 192.168.20.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.20.2, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) ``` Traffic does appear to be encapsulated correctly to the destination so it seems to be an issue with the receiving router decapsulating the instruction. The following was observed on *ubnt2* receiving pings from PC1 via *ubnt1*: ```linux root@ubnt2:~# tcpdump -i ens3 host 2001:db8:2:0:100:: tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 17:51:41.342725 IP6 fd9b:ee89:41b5:edcc:c799:9317:acda:ebf1 > 2001:db8:2:0:100::: srcrt (len=2, type=4, segleft=0[|srcrt] 17:51:43.340045 IP6 fd9b:ee89:41b5:edcc:c799:9317:acda:ebf1 > 2001:db8:2:0:100::: srcrt (len=2, type=4, segleft=0[|srcrt] 17:51:45.341147 IP6 fd9b:ee89:41b5:edcc:c799:9317:acda:ebf1 > 2001:db8:2:0:100::: srcrt (len=2, type=4, segleft=0[|srcrt] 17:51:47.341180 IP6 fd9b:ee89:41b5:edcc:c799:9317:acda:ebf1 > 2001:db8:2:0:100::: srcrt (len=2, type=4, segleft=0[|srcrt] 17:51:49.342127 IP6 fd9b:ee89:41b5:edcc:c799:9317:acda:ebf1 > 2001:db8:2:0:100::: srcrt (len=2, type=4, segleft=0[|srcrt] ``` No `echo-request` is seen on **ubnt2** exiting the VRF142 interface (ens4) towards PC2. ## FRR SRv6 End.DT4 troubleshooting Examining the Zebra log revealed the following: ```linux 2022/06/27 19:16:59 ZEBRA: [G08K1-416PW] netlink_nexthop_msg_encode: unsupport seg6local behaviour action=8 2022/06/27 19:16:59 ZEBRA: [G08K1-416PW] netlink_nexthop_msg_encode: unsupport seg6local behaviour action=8 2022/06/27 19:16:59 ZEBRA: [NZNZ4-7P54Y] default(0:254):2001:db8:1:0:100::/128: Processing rn 0x55747750b5d0 2022/06/27 19:16:59 ZEBRA: [ZJVZ4-XEGPF] default(0:254):2001:db8:1:0:100::/128: Examine re 0x55747750b050 (bgp) status: Changed flags: Recursion dist 20 metric 0 2022/06/27 19:16:59 ZEBRA: [JPJF4-TGCY5] default(0:254):2001:db8:1:0:100::/128: After processing: old_selected 0x0 new_selected 0x55747750b050 old_fib 0x0 new_fib 0x55747750b050 2022/06/27 19:16:59 ZEBRA: [ZFRN1-EZNJZ] default(0:254):2001:db8:1:0:100::/128: Adding route rn 0x55747750b5d0, re 0x55747750b050 (bgp) 2022/06/27 19:16:59 ZEBRA: [QZ1V6-CRT8D] default(0:254):2001:db8:1:0:100::/128 rn 0x55747750b5d0 dequeued from sub-queue 6 2022/06/27 19:16:59 ZEBRA: [NZNZ4-7P54Y] default(0:254):2001:db8:1:0:200::/128: Processing rn 0x55747750b950 2022/06/27 19:16:59 ZEBRA: [ZJVZ4-XEGPF] default(0:254):2001:db8:1:0:200::/128: Examine re 0x55747750abc0 (bgp) status: Changed flags: Recursion dist 20 metric 0 2022/06/27 19:16:59 ZEBRA: [JPJF4-TGCY5] default(0:254):2001:db8:1:0:200::/128: After processing: old_selected 0x0 new_selected 0x55747750abc0 old_fib 0x0 new_fib 0x55747750abc0 2022/06/27 19:16:59 ZEBRA: [ZFRN1-EZNJZ] default(0:254):2001:db8:1:0:200::/128: Adding route rn 0x55747750b950, re 0x55747750abc0 (bgp) 2022/06/27 19:16:59 ZEBRA: [QZ1V6-CRT8D] default(0:254):2001:db8:1:0:200::/128 rn 0x55747750b950 dequeued from sub-queue 6 2022/06/27 19:16:59 ZEBRA: [P2XBZ-RAFQ5][EC 4043309074] Failed to install Nexthop ID (76) into the kernel 2022/06/27 19:16:59 ZEBRA: [HSYZM-HV7HF] Extended Error: Nexthop id does not exist 2022/06/27 19:16:59 ZEBRA: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=32, pid=3300413719 2022/06/27 19:16:59 ZEBRA: [VYKYC-709DP] default(0:254):2001:db8:1:0:100::/128: Route install failed ``` ## Summary SRv6 brings a lot of new potential and simplification of existing services that are offered via MPLS or derivatives. Although there are stumbling blocks at present, at least in the Linux world, I believe there is a bright future for SRv6. I hope to update this post in the future once issues with the `End.DT4` endpoint function are resolved. **UPDATE 12-17-2024**: I [tried](/2024/12/16/srv6-frr) a similar configuration with Debian 12 and FRR 10.2 that works! ## References * [SRv6 Introduction](https://www.segment-routing.net/tutorials/2017-12-05-srv6-introduction/) * [FRR Pull Request #9649 (add support for SRv6 IPv4 L3VPN)](https://github.com/FRRouting/frr/pull/9649) * [seg6: add support for the SRv6 End.DT4 behavior](https://lkml.org/lkml/2020/11/23/1439) * [PoC: FRRouting SRv6 L3VPN (IPv4 and IPv6)](https://www.linkedin.com/pulse/poc-frrouting-srv6-l3vpn-ipv4-ipv6and-wireguard-vpn-gonzalez-diaz) * [Creating a VRF and Running Services Inside it on Linux](https://interpip.es/linux/creating-a-vrf-and-running-services-inside-it-on-linux/)