# Segment Routing IPv6 (SRv6) with VyOS In this post we take a look at configuring Segment Routing IPv6 (SRv6) using [VyOS](https://vyos.net/). For those unfamiliar, VyOS is an open-source routing platform built on top of Debian Linux and features a commit-based configuration system similar to Juniper's JUNOS. It originated as a fork of Vyatta Core in late 2013 after Brocade stopped development. If you are new to VyOS, it is recommended to read the [quick start](https://docs.vyos.io/en/latest/quick-start.html) guide before continuing. Beginning with pull request [#2606](https://github.com/vyos/vyos-1x/pull/2606), the VyOS team has made several strides to include configuration options for [FRR's](https://frrouting.org/) BGP SRv6 capabilities. FRR is the routing control plane used in VyOS. This post looks specifically at configuring SRv6 L3VPNs. {{< admonition type=warning >}} It is important to note that while this article will result in a working SRv6 L3VPN network, there are a few caveats (~~[1](#pe-caveat)~~, [2](#caveat)) as of `VyOS 1.5-rolling-202401310023` {{< /admonition >}} ## Topology ![Topology Diagram](/posts/2024/02/srv6-vyos/topology.png) Above is a very basic service provider topology. On the left is our first customer (CE1). It connects to the network using a standard dual-stack IPv4/IPv6 ethernet connection to the provider network at (PE2). Similarly, the right side of the diagram is the other end of the customer VPN (CE5). It has an identical configuration as CE1, updated with its own addressing, and connects to PE4. The service provider network consists of two provider-edge (PE) routers and one provider (P) router. The P router (P3) is completely unnecessary and could be removed at any point. It only serves to demonstrate that there is no special connectivity needed between PE routers other than basic IPv6 connectivity. The underlay within the provider network uses IS-IS between unnumbered ethernet interfaces, relying on IPv6 link-local. There are no addresses configured at all on P3 to demonstrate this simplicity. Each PE router advertises its assigned /48 IPv6 summary into IS-IS as a means for all routers (including P3) to understand how to route traffic within these prefixes to each PE. *There are no IPv4 elements within the service provider underlay network (AS65000) at all*. Between provider-edge routers, iBGP is configured with both IPv4 and IPv6 VPN address-families. This provides for the exchange of L3VPN routes for each customer using a mechanism similar to more traditional methods such as MPLS L3VPN. ## Configuration Everything discussed and demonstrated here has complete configurations as well as the pre-configured lab for EVE-NG available on [GitHub](https://github.com/jvoss/eve-ng-labs/tree/main/service_provider/srv6/vyos). The VyOS version used is a [nightly build](https://vyos.net/get/nightly-builds/) from the end of January 2024 (`VyOS 1.5-rolling-202401310023`). ### Provider Routers #### PE2 Configure the host name: ``` set system host-name 'PE2' ``` ##### Interfaces The provider network interface configuration (`eth0`) is trivial. There is nothing special needed here other than to ensure the interface is not disabled and has a IPv6 link-local address (both defaults). The description is set here as a standard best practice. Additionally, a routable IPv6 loopback addresses is configured for use as a destination address for BGP sessions between PE routers. ``` set interfaces ethernet eth0 description 'P3' set interfaces loopback lo address '2001:db8:2:ffff::2/128' ``` ##### Routing A static IPv6 null route is configured to create a summary which encompasses all future assignments related to this router. In our case, this will cover Segment Identifiers (SIDs) represented as IPv6 addresses. ``` set protocols static route6 2001:db8:2::/48 blackhole ``` ###### IS-IS IS-IS is configured to provide the underlay routing for all provider routers. This requires enabling the protocol on interfaces `eth0` and `lo`, setting the device to act as a Level-2 router and configuring a Network Entity Title (NET). ``` set protocols isis interface eth0 set protocols isis interface lo set protocols isis level 'level-2' set protocols isis net '49.0000.2222.2222.2222.00' ``` Additionally, we redistribute the IPv6 summary we created above. This informs the intermediary P router, and other PEs, how to route traffic destined for addresses within this PE's assigned summary (including soon to be created SRv6 SIDs). ``` set protocols isis redistribute ipv6 static level-2 ``` ###### Segment Routing Now it is time to configure the basic system properties for Segment Routing IPv6. Here we enable the interfaces that will receive SRv6 traffic (our provider network interfaces). We also must define at least one prefix that can be used to allocate Segment IDs (SIDs). This is known as the SRv6 Locator and is being named `L3VPN`. Optionally, we can also configure the [micro-sid](https://www.ietf.org/archive/id/draft-filsfils-spring-net-pgm-extension-srv6-usid-16.txt) allocation behavior. ``` set protocols segment-routing interface eth0 srv6 set protocols segment-routing srv6 locator L3VPN prefix '2001:db8:2:aaa::/64' set protocols segment-routing srv6 locator L3VPN behavior-usid ``` ###### BGP At this point we can now configure BGP. First, configure the autonomous system number `65000`. Next, since there are no IPv4 addresses configured we must supply a router ID. Note that while required to be in IPv4 format, this address does not have to exist on any interface. ``` set protocols bgp system-as '65000' set protocols bgp parameters router-id '10.0.0.2' ``` Now as a matter of practice, configure a peer-group that can be used to templatize configuration of multiple PEs. In this lab we have only one other PE neighbor, however it is being done here to more clearly show the necessary attributes for PE neighbors. The `PE` peer group configures both IPv4 VPN and IPv6 VPN address families which provides the mechanism to exchange IPv4 and IPv6 customer (VRF) routes between PE routers. In this scenario we are using iBGP between PE routers and configure the peers to match the "internal" ASN (`65000`) we configured above. ``` set protocols bgp peer-group PE address-family ipv4-vpn nexthop-self set protocols bgp peer-group PE address-family ipv6-vpn nexthop-self set protocols bgp peer-group PE remote-as 'internal' set protocols bgp peer-group PE update-source 'lo' ``` Next we need to configure the IPv6 nexthop capability. [RFC 8950](https://www.rfc-editor.org/rfc/rfc8950) describes this mechanism as a way to encode an IPv6 address as a next-hop for IPv4 destinations. This is required in order for the L3VPN to function in Linux for customer IPv4 prefixes. As we will see later, customer IPv4 routes will be destined to the exit PE's IPv6 address along with an assigned IPv6 SID. ``` set protocols bgp peer-group PE capability extended-nexthop ``` We can now configure the `PE4` router as a member of the `PE` peer-group. ``` set protocols bgp neighbor 2001:db8:4:ffff::4 description PE4 set protocols bgp neighbor 2001:db8:4:ffff::4 peer-group 'PE' ``` The last step for now is to configure the SRv6 locator that BGP should use to generate SIDs. Here we configure the locator named `L3VPN` we created previously. ``` set protocols bgp srv6 locator 'L3VPN' ``` ##### Customer Configuration ###### VRF First define a VRF to contain the customer's routes (CE1). This lab uses the name `VRF1`. Linux also requires a table ID to identify the routing table: ``` set vrf name VRF1 table '101' ``` ###### Interfaces Next configure the interface connected to CE1 and place it into the VRF: ``` set interfaces ethernet eth1 address '172.16.12.2/24' set interfaces ethernet eth1 address 'fd01:101:12::2/64' set interfaces ethernet eth1 description 'CE1' set interfaces ethernet eth1 vrf 'VRF1' ``` ###### BGP In this lab, the CE routers have the ability to advertise any additional private customer routes that need to be shared between locations. To accomplish this, eBGP is configured with in the VRF. First set a `router-id` and `system-as` for the VRF's BGP instance: ``` set vrf name VRF1 protocols bgp parameters router-id '10.0.0.2' set vrf name VRF1 protocols bgp system-as '65000' ``` Next, configure both IPv4 and IPv6 address families. For the purposes of this lab, configure redistribution of connected routes. ``` set vrf name VRF1 protocols bgp address-family ipv4-unicast redistribute connected set vrf name VRF1 protocols bgp address-family ipv6-unicast redistribute connected ``` Now configure a peer-group (`CE`) to define the BGP peering configuration for our CE routers. Here we enable both IPv4 and IPv6 unicast addresses families and instruct the group to be an eBGP session. In this example, any AS that does not match our `system-as` will be accepted, using the `external` keyword. ``` set vrf name VRF1 protocols bgp peer-group CE address-family ipv4-unicast set vrf name VRF1 protocols bgp peer-group CE address-family ipv6-unicast set vrf name VRF1 protocols bgp peer-group CE remote-as 'external' ``` Once the peer-group is defined, we can now add `CE1` as a neighbor. ``` set vrf name VRF1 protocols bgp neighbor fd01:101:12::1 peer-group 'CE' ``` ###### BGP VPN There are additional parameters we need to configure to identify, export and import customer routes between VRF1 and the global BGP IPv4/IPv6 VPN table. First is the [route distinguisher](https://en.wikipedia.org/wiki/Route_distinguisher). It uniquely identifies a route within the global IPv4 and IPv6 VPN tables and allows for address overlapping within multiple VRFs. In this case we are using `:` as a scheme. It needs to be configured on both IPv4 and IPv6 address families inside `VRF1`. ``` set vrf name VRF1 protocols bgp address-family ipv4-unicast rd vpn export '10.0.0.2:101' set vrf name VRF1 protocols bgp address-family ipv6-unicast rd vpn export '10.0.0.2:101' ``` Next we need to define route targets. These extended communities are used to identify which routes to import from the global IPv4/IPv6 VPN tables and how to mark routes being sent to the global IPv4/IPv6 VPN tables respectively. ``` set vrf name VRF1 protocols bgp address-family ipv4-unicast route-target vpn export '65000:101' set vrf name VRF1 protocols bgp address-family ipv4-unicast route-target vpn import '65000:101' set vrf name VRF1 protocols bgp address-family ipv6-unicast route-target vpn export '65000:101' set vrf name VRF1 protocols bgp address-family ipv6-unicast route-target vpn import '65000:101' ``` {{< admonition type=note >}} A shortcut option `both` does exist. Instead of the above the same can be accomplished using: ``` set vrf name VRF1 protocols bgp address-family ipv4-unicast route-target vpn both '65000:101' set vrf name VRF1 protocols bgp address-family ipv6-unicast route-target vpn both '65000:101' ``` {{< /admonition >}} We can now instruct FRR to perform importing and exporting routes to the global IPv4/IPv6 VPN tables: ``` set vrf name VRF1 protocols bgp address-family ipv4-unicast export vpn set vrf name VRF1 protocols bgp address-family ipv4-unicast import vpn set vrf name VRF1 protocols bgp address-family ipv6-unicast export vpn set vrf name VRF1 protocols bgp address-family ipv6-unicast import vpn ``` Finally, SRv6 is configured for the VRF. Set the SID creation option to `per-vrf`, creating a unique SID per VRF. The `export '101'` statement assigns a specific allocation index within the locator prefix for this VRF. ``` set vrf name VRF1 protocols bgp sid vpn per-vrf export '101' ``` When using a specific index we can predict a resulting SID. In this case `101` is represented as `65` in hexadecimal and produces the SID: `2001:db8:2:aaa:65::/128` {{< admonition type=note >}} Automatic allocation indexing is also an option. Use the following instead of a specific index number: ``` set vrf name VRF1 protocols bgp sid vpn per-vrf export 'auto' ``` {{< /admonition >}} Prior to `VyOS 1.5-rolling-202402080022` a **reboot** is required. This was necessary to ensure that the `net.vrf.strict_mode` parameter gets set to `1`. See the caveat details below. ##### PE Caveat {{< admonition type=bug >}} **UPDATE**: This has been resolved as of `VyOS 1.5-rolling-202402080022` via [VyOS#2952](https://github.com/vyos/vyos-1x/pull/2952). The workaround mentioned here is no longer necessary on newer versions. VyOS PRs [#2663](https://github.com/vyos/vyos-1x/pull/2663) and [#2877](https://github.com/vyos/vyos-1x/pull/2877) attempt to set the kernel parameter `net.vrf.strict_mode=1` automatically for you. This parameter is required for SRv6 to function and must be set on both PE routers. After initial configuration a reboot is required in order for this to take effect. However, after restarting, it does not appear to get correctly set: ``` vyos@PE2:~$ sysctl net.vrf.strict_mode net.vrf.strict_mode = 0 ``` This means FRR is unable to install SRv6 SIDs into the Linux kernel routing table: ``` vyos@PE2:~$ show ipv6 route bgp Codes: K - kernel route, C - connected, S - static, R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure B>r 2001:db8:2:aaa:65::/128 [20/0] is directly connected, VRF1, seg6local End.DT46 table 101, weight 1, 00:40:55 ``` Notice the above prefix shows a `r` (rejected) status. The following log entries confirm the issue installing into kernel: ``` Feb 01 16:58:47 PE2 zebra[1264]: [X5XE1-RS0SW][EC 4043309074] Failed to install Nexthop (14[if 7]) into the kernel Feb 01 16:58:47 PE2 zebra[1264]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=12, pid=2945720226 Feb 01 16:58:47 PE2 zebra[1264]: [HSYZM-HV7HF] Extended Error: Nexthop id does not exist Feb 01 16:58:47 PE2 zebra[1264]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: Operation not permitted, type=RTM_NEWNEXTHOP(104), seq=11, pid=2945720226 Feb 01 16:58:47 PE2 zebra[1264]: [HSYZM-HV7HF] Extended Error: Strict mode for VRF is disabled ``` **WORKAROUND**: I have observed that manually setting this parameter in the configuration (committing and saving) and *rebooting* again resolves this issue: ``` set system sysctl parameter net.vrf.strict_mode value '1' ``` Be sure the other changes are committed before attempting to apply this parameter. Otherwise you may experience a `Invalid value` error. After one last reboot, the SID appears to have been programmed correctly: ``` vyos@PE2:~$ show ipv6 route bgp Codes: K - kernel route, C - connected, S - static, R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure B>* 2001:db8:2:aaa:65::/128 [20/0] is directly connected, VRF1, seg6local End.DT46 table 101, weight 1, 00:01:18 ``` {{< /admonition >}} #### P3 The P router (`P3`) requires very minimal configuration. The only requirement is to enable IS-IS on both ethernet interfaces and optionally set a few descriptions. * [P3 complete configuration](https://github.com/jvoss/eve-ng-labs/blob/main/service_provider/srv6/vyos/P3/config.commands) ``` set system host-name 'P3' ``` ##### Interfaces ``` set interfaces ethernet eth0 description 'PE2' set interfaces ethernet eth1 description 'PE4' ``` ##### IS-IS ``` set protocols isis interface eth0 set protocols isis interface eth1 set protocols isis level 'level-2' set protocols isis net '49.0000.3333.3333.3333.00' ``` #### PE4 `PE4` is configured identically to [PE2](#pe2). A complete configuration can be found on [GitHub](https://github.com/jvoss/eve-ng-labs/blob/main/service_provider/srv6/vyos/PE4/config.commands) and specific differences are highlighted below below: ``` set system host-name 'PE4' ``` ##### Interfaces ``` set interfaces loopback lo address '2001:db8:4:ffff::4/128' ``` ##### Routing ``` set protocols static route6 2001:db8:4::/48 blackhole ``` ###### IS-IS ``` set protocols isis net '49.0000.4444.4444.4444.00' ``` ###### Segment Routing ``` set protocols segment-routing srv6 locator L3VPN prefix '2001:db8:4:aaa::/64' ``` ###### BGP ``` set protocols bgp parameters router-id '10.0.0.4' set protocols bgp neighbor 2001:db8:2:ffff::2 description PE2 set protocols bgp neighbor 2001:db8:2:ffff::2 peer-group 'PE' ``` ##### Customer Configuration ###### Interfaces ``` set interfaces ethernet eth1 address '172.16.45.4/24' set interfaces ethernet eth1 address 'fd01:101:2d::4/64' set interfaces ethernet eth1 description 'CE2' set interfaces ethernet eth1 vrf 'VRF1' ``` ###### BGP ``` set vrf name VRF1 protocols bgp parameters router-id '10.0.0.4' set vrf name VRF1 protocols bgp neighbor fd01:101:2d::5 peer-group 'CE' ``` ###### BGP VPN ``` set vrf name VRF1 protocols bgp address-family ipv4-unicast rd vpn export '10.0.0.4:101' set vrf name VRF1 protocols bgp address-family ipv6-unicast rd vpn export '10.0.0.4:101' ``` ### Customer Routers CE routers only require basic configuration. Substitute addresses and values for each router according to the [topology diagram](#topology): * [CE1 configuration](https://github.com/jvoss/eve-ng-labs/blob/main/service_provider/srv6/vyos/CE1/config.commands) * [CE5 configuration](https://github.com/jvoss/eve-ng-labs/blob/main/service_provider/srv6/vyos/CE5/config.commands) ``` set system host-name 'CE1' ``` #### Interfaces ``` set interfaces ethernet eth0 description 'PE2' set interfaces ethernet eth0 address '172.16.12.1/24' set interfaces ethernet eth0 address 'fd01:101:12::1/64' ``` #### BGP ``` set protocols bgp system-as '65001' set protocols bgp address-family ipv4-unicast set protocols bgp address-family ipv6-unicast set protocols bgp neighbor fd01:101:12::2 address-family ipv4-unicast set protocols bgp neighbor fd01:101:12::2 address-family ipv6-unicast set protocols bgp neighbor fd01:101:12::2 remote-as 'external' ``` ## Verification The following can be used to validate the control plane in each PE router. ### SRv6 SIDs Verify a SRv6 `End.DT46` SID has been created for VRF1 (`table 101`): ``` vyos@PE2:~$ show ipv6 route bgp Codes: K - kernel route, C - connected, S - static, R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure B>* 2001:db8:2:aaa:65::/128 [20/0] is directly connected, VRF1, seg6local End.DT46 table 101, weight 1, 00:00:21 ``` More information about SR Endpoint behaviors can be found in [RFC 8986](https://www.rfc-editor.org/rfc/rfc8986.html#name-sr-endpoint-behaviors) ### BGP **IPv4: VPN** Validate routes are in the global IPv4 VPN table: ``` vyos@PE2:~$ show bgp ipv4 vpn BGP table version is 1, local router ID is 10.0.0.2, vrf id 0 Default local pref 100, local AS 65000 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 10.0.0.2:101 *> 172.16.12.0/24 0.0.0.0@7< 0 32768 ? UN=0.0.0.0 EC{65000:101} label=1616 sid=2001:db8:2:aaa:: sid_structure=[40,24,16,0] type=bgp, subtype=5 Route Distinguisher: 10.0.0.4:101 *>i172.16.45.0/24 2001:db8:4:ffff::4 0 100 0 ? UN=2001:db8:4:ffff::4 EC{65000:101} label=1616 sid=2001:db8:4:aaa:: sid_structure=[40,24,16,0] type=bgp, subtype=0 Displayed 2 routes and 2 total paths ``` **IPv6: VPN** Validate routes are in the global IPv6 VPN table: ``` vyos@PE2:~$ show bgp ipv6 vpn BGP table version is 1, local router ID is 10.0.0.2, vrf id 0 Default local pref 100, local AS 65000 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 10.0.0.2:101 *> fd01:101:12::/64 ::@7< 0 32768 ? UN=:: EC{65000:101} label=1616 sid=2001:db8:2:aaa:: sid_structure=[40,24,16,0] type=bgp, subtype=5 Route Distinguisher: 10.0.0.4:101 *>ifd01:101:2d::/64 2001:db8:4:ffff::4 0 100 0 ? UN=2001:db8:4:ffff::4 EC{65000:101} label=1616 sid=2001:db8:4:aaa:: sid_structure=[40,24,16,0] type=bgp, subtype=0 Displayed 2 routes and 2 total paths ``` **IPv4: VRF1** Validate IPv4 routes are received from the CE and are imported into the VRF from other PEs: ``` vyos@PE2:~$ show bgp vrf VRF1 ipv4 BGP table version is 2, local router ID is 10.0.0.2, vrf id 7 Default local pref 100, local AS 65000 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *> 172.16.12.0/24 0.0.0.0 0 32768 ? *> 172.16.45.0/24 2001:db8:4:ffff::4@0< 0 100 0 ? Displayed 2 routes and 2 total paths ``` **IPv6: VRF1** Validate IPv6 routes are received from the CE and are imported into the VRF from other PEs: ``` vyos@PE2:~$ show bgp vrf VRF1 ipv6 BGP table version is 2, local router ID is 10.0.0.2, vrf id 7 Default local pref 100, local AS 65000 Status codes: s suppressed, d damped, h history, * valid, > best, = multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *> fd01:101:12::/64 :: 0 32768 ? *> fd01:101:2d::/64 2001:db8:4:ffff::4@0< 0 100 0 ? Displayed 2 routes and 2 total paths ``` ### Routing Tables Verify routes are installed into the Linux kernel for the VRF routing table: **IPv4** ``` vyos@PE2:~$ show ip route vrf VRF1 Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF VRF1: C>* 172.16.12.0/24 is directly connected, eth1, 00:13:13 B> 172.16.45.0/24 [200/0] via 2001:db8:4:ffff::4 (vrf default) (recursive), label 1616, seg6 2001:db8:4:aaa:65::, weight 1, 00:12:31 * via fe80::5201:ff:fe03:0, eth0 (vrf default), label 1616, seg6 2001:db8:4:aaa:65::, weight 1, 00:12:31 ``` **IPv6** ``` vyos@PE2:~$ show ipv6 route vrf VRF1 Codes: K - kernel route, C - connected, S - static, R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backup t - trapped, o - offload failure VRF VRF1: C>* fd01:101:12::/64 is directly connected, eth1, 00:13:36 B> fd01:101:2d::/64 [200/0] via 2001:db8:4:ffff::4 (vrf default) (recursive), label 1616, seg6 2001:db8:4:aaa:65::, weight 1, 00:12:56 * via fe80::5201:ff:fe03:0, eth0 (vrf default), label 1616, seg6 2001:db8:4:aaa:65::, weight 1, 00:12:56 C>* fe80::/64 is directly connected, eth1, 00:13:37 ``` ### End-to-End connectivity #### Caveat {{< admonition type=bug >}} **UPDATE**: A workaround to this bug is discussed [here](/2024/12/26/srv6-vyos-revisit). Before any L3VPN traffic will route properly, traffic must be sent sourced from either *PE* to the other *PE*, *within the VRF* at least once. **This must be performed specifically between VRF-member interfaces on PEs.** For example, a ping between PE2's `eth1` interface and PE4's `eth1` interface: ``` vyos@PE2:~$ ping 172.16.45.4 source-address 172.16.12.2 vrf VRF1 PING 172.16.45.5 (172.16.45.5) from 172.16.12.2 : 56(84) bytes of data. 64 bytes from 172.16.45.5: icmp_seq=1 ttl=64 time=1.90 ms 64 bytes from 172.16.45.5: icmp_seq=2 ttl=64 time=1.42 ms ``` This only needs to be performed once (unless rebooted) and can be completed on either address family. It does not require this action on both families in order for both IPv4 and IPv6 VRF traffic to work. Once completed, all further traffic works as expected. This is a particularly odd behavior that appears to have something to do with routing within the Linux kernel. Using [dropwatch](https://github.com/nhorman/dropwatch) on the inbound PE and collecting dropped packets from CE1 to CE5 (on PE2) I observed the following: ``` # PE2 drop at: ip6_pkt_drop+0xeb/0x1c0 (0xffffffff9512123b) origin: software input port ifindex: 7 timestamp: Sun Feb 4 19:10:30 2024 028938864 nsec protocol: 0x86dd length: 162 original length: 162 drop reason: IP_INNOROUTES ``` The input port (`ifindex: 7`) matches the `VRF1` interface. The drop reason listed, [IP_INNOROUTES](https://github.com/torvalds/linux/blob/54be6c6c5ae8e0d93a6c4641cb7528eb0b6ba478/include/net/dropreason-core.h#L345), corresponds to a [network unreachable error](https://lore.kernel.org/lkml/20220413081600.187339-10-imagedong@tencent.com/T/). These drops are no longer observed after a successful ping between PE2 and PE4's `eth1` interfaces as shown above. {{< /admonition >}} #### IPv4 Ping from CE1 to CE5 ``` vyos@CE1:~$ ping 172.16.45.5 PING 172.16.45.5 (172.16.45.5) 56(84) bytes of data. 64 bytes from 172.16.45.5: icmp_seq=1 ttl=63 time=2.95 ms 64 bytes from 172.16.45.5: icmp_seq=2 ttl=63 time=1.88 ms 64 bytes from 172.16.45.5: icmp_seq=3 ttl=63 time=2.24 ms 64 bytes from 172.16.45.5: icmp_seq=4 ttl=63 time=1.95 ms 64 bytes from 172.16.45.5: icmp_seq=5 ttl=63 time=1.89 ms ^C --- 172.16.45.5 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms rtt min/avg/max/mdev = 1.878/2.181/2.954/0.407 ms ``` The packet capture below shows the traffic above collected on PE2's `eth0` interface. This is the interface from PE2 into the provider network facing `P3`. ![ICMP Packet Capture](/posts/2024/02/srv6-vyos/icmp_ce1_ce5_pe2_eth0.png) Notice the IPv4 traffic is encapsulated into IPv6 with a destination address of the `End.DT46` [SID allocated](#srv6-sids) for `VRF1` (`table 101`) on PE4. #### IPv6 Ping from CE1 to CE5 ``` vyos@CE1:~$ ping fd01:101:2d::5 PING fd01:101:2d::5(fd01:101:2d::5) 56 data bytes 64 bytes from fd01:101:2d::5: icmp_seq=1 ttl=63 time=2.68 ms 64 bytes from fd01:101:2d::5: icmp_seq=2 ttl=63 time=1.88 ms 64 bytes from fd01:101:2d::5: icmp_seq=3 ttl=63 time=1.92 ms 64 bytes from fd01:101:2d::5: icmp_seq=4 ttl=63 time=2.13 ms 64 bytes from fd01:101:2d::5: icmp_seq=5 ttl=63 time=2.08 ms ^C --- fd01:101:2d::5 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4005ms rtt min/avg/max/mdev = 1.878/2.138/2.684/0.289 ms ``` This packet capture again shows the traffic above collected on PE2, `eth0`. ![ICMP Packet Capture](/posts/2024/02/srv6-vyos/icmp_ce1_ce5_pe2_eth0.png) Similarly to IPv4 test above, notice the traffic is encapsulated with the same destination address representing the `End.DT46` [SID allocated](#srv6-sids) for `VRF1` (`table 101`) on PE4. ## Conclusion IPv6 as a transport mechanism for Segment Routing allows for more traditional provider topologies to be simplified significantly as long as there is basic IPv6 connectivity between customer facing routers. In this lab we created a basic working SRv6 L3VPN network in VyOS. We walked briefly through all of the necessary components and took a peak at how to validate end-to-end connectivity. I hope this was beneficial in providing some background on Segment Routing with IPv6 and how this technology can be used to support Layer 3 VPNs.