Segment Routing IPv6 (SRv6) with VyOS

1 Feb 2024 4164 words 20 minutes

Contents

In this post we take a look at configuring Segment Routing IPv6 (SRv6) using VyOS. For those unfamiliar, VyOS is an open-source routing platform built on top of Debian Linux and features a commit-based configuration system similar to Juniper’s JUNOS. It originated as a fork of Vyatta Core in late 2013 after Brocade stopped development. If you are new to VyOS, it is recommended to read the quick start guide before continuing.

Beginning with pull request #2606, the VyOS team has made several strides to include configuration options for FRR’s BGP SRv6 capabilities. FRR is the routing control plane used in VyOS.

This post looks specifically at configuring SRv6 L3VPNs.

Warning

It is important to note that while this article will result in a working SRv6 L3VPN network, there are a few caveats (1, 2) as of VyOS 1.5-rolling-202401310023

Topology

Above is a very basic service provider topology. On the left is our first customer (CE1). It connects to the network using a standard dual-stack IPv4/IPv6 ethernet connection to the provider network at (PE2). Similarly, the right side of the diagram is the other end of the customer VPN (CE5). It has an identical configuration as CE1, updated with its own addressing, and connects to PE4.

The service provider network consists of two provider-edge (PE) routers and one provider (P) router. The P router (P3) is completely unnecessary and could be removed at any point. It only serves to demonstrate that there is no special connectivity needed between PE routers other than basic IPv6 connectivity.

The underlay within the provider network uses IS-IS between unnumbered ethernet interfaces, relying on IPv6 link-local. There are no addresses configured at all on P3 to demonstrate this simplicity. Each PE router advertises its assigned /48 IPv6 summary into IS-IS as a means for all routers (including P3) to understand how to route traffic within these prefixes to each PE. There are no IPv4 elements within the service provider underlay network (AS65000) at all.

Between provider-edge routers, iBGP is configured with both IPv4 and IPv6 VPN address-families. This provides for the exchange of L3VPN routes for each customer using a mechanism similar to more traditional methods such as MPLS L3VPN.

Configuration

Everything discussed and demonstrated here has complete configurations as well as the pre-configured lab for EVE-NG available on GitHub. The VyOS version used is a nightly build from the end of January 2024 (VyOS 1.5-rolling-202401310023).

Provider Routers

PE2

Configure the host name:

1

set system host-name 'PE2'

Interfaces

The provider network interface configuration (eth0) is trivial. There is nothing special needed here other than to ensure the interface is not disabled and has a IPv6 link-local address (both defaults). The description is set here as a standard best practice.

Additionally, a routable IPv6 loopback addresses is configured for use as a destination address for BGP sessions between PE routers.

1
2


set interfaces ethernet eth0 description 'P3'
set interfaces loopback lo address '2001:db8:2:ffff::2/128'

Routing

A static IPv6 null route is configured to create a summary which encompasses all future assignments related to this router. In our case, this will cover Segment Identifiers (SIDs) represented as IPv6 addresses.

1

set protocols static route6 2001:db8:2::/48 blackhole

IS-IS

IS-IS is configured to provide the underlay routing for all provider routers. This requires enabling the protocol on interfaces eth0 and lo, setting the device to act as a Level-2 router and configuring a Network Entity Title (NET).

1
2
3
4


set protocols isis interface eth0
set protocols isis interface lo
set protocols isis level 'level-2'
set protocols isis net '49.0000.2222.2222.2222.00'

Additionally, we redistribute the IPv6 summary we created above. This informs the intermediary P router, and other PEs, how to route traffic destined for addresses within this PE’s assigned summary (including soon to be created SRv6 SIDs).

1

set protocols isis redistribute ipv6 static level-2

Segment Routing

Now it is time to configure the basic system properties for Segment Routing IPv6. Here we enable the interfaces that will receive SRv6 traffic (our provider network interfaces). We also must define at least one prefix that can be used to allocate Segment IDs (SIDs). This is known as the SRv6 Locator and is being named L3VPN. Optionally, we can also configure the micro-sid allocation behavior.

1
2
3


set protocols segment-routing interface eth0 srv6
set protocols segment-routing srv6 locator L3VPN prefix '2001:db8:2:aaa::/64'
set protocols segment-routing srv6 locator L3VPN behavior-usid

BGP

At this point we can now configure BGP. First, configure the autonomous system number 65000. Next, since there are no IPv4 addresses configured we must supply a router ID. Note that while required to be in IPv4 format, this address does not have to exist on any interface.

1
2


set protocols bgp system-as '65000'
set protocols bgp parameters router-id '10.0.0.2'

Now as a matter of practice, configure a peer-group that can be used to templatize configuration of multiple PEs. In this lab we have only one other PE neighbor, however it is being done here to more clearly show the necessary attributes for PE neighbors.

The PE peer group configures both IPv4 VPN and IPv6 VPN address families which provides the mechanism to exchange IPv4 and IPv6 customer (VRF) routes between PE routers. In this scenario we are using iBGP between PE routers and configure the peers to match the “internal” ASN (65000) we configured above.

1
2
3
4


set protocols bgp peer-group PE address-family ipv4-vpn nexthop-self
set protocols bgp peer-group PE address-family ipv6-vpn nexthop-self
set protocols bgp peer-group PE remote-as 'internal'
set protocols bgp peer-group PE update-source 'lo'

Next we need to configure the IPv6 nexthop capability. RFC 8950 describes this mechanism as a way to encode an IPv6 address as a next-hop for IPv4 destinations. This is required in order for the L3VPN to function in Linux for customer IPv4 prefixes. As we will see later, customer IPv4 routes will be destined to the exit PE’s IPv6 address along with an assigned IPv6 SID.

1

set protocols bgp peer-group PE capability extended-nexthop

We can now configure the PE4 router as a member of the PE peer-group.

1
2


set protocols bgp neighbor 2001:db8:4:ffff::4 description PE4
set protocols bgp neighbor 2001:db8:4:ffff::4 peer-group 'PE'

The last step for now is to configure the SRv6 locator that BGP should use to generate SIDs. Here we configure the locator named L3VPN we created previously.

1

set protocols bgp srv6 locator 'L3VPN'

Customer Configuration

VRF

First define a VRF to contain the customer’s routes (CE1). This lab uses the name VRF1. Linux also requires a table ID to identify the routing table:

1

set vrf name VRF1 table '101'

Interfaces

Next configure the interface connected to CE1 and place it into the VRF:

1
2
3
4


set interfaces ethernet eth1 address '172.16.12.2/24'
set interfaces ethernet eth1 address 'fd01:101:12::2/64'
set interfaces ethernet eth1 description 'CE1'
set interfaces ethernet eth1 vrf 'VRF1'

BGP

In this lab, the CE routers have the ability to advertise any additional private customer routes that need to be shared between locations. To accomplish this, eBGP is configured with in the VRF.

First set a router-id and system-as for the VRF’s BGP instance:

1
2


set vrf name VRF1 protocols bgp parameters router-id '10.0.0.2'
set vrf name VRF1 protocols bgp system-as '65000'

Next, configure both IPv4 and IPv6 address families. For the purposes of this lab, configure redistribution of connected routes.

1
2


set vrf name VRF1 protocols bgp address-family ipv4-unicast redistribute connected
set vrf name VRF1 protocols bgp address-family ipv6-unicast redistribute connected

Now configure a peer-group (CE) to define the BGP peering configuration for our CE routers. Here we enable both IPv4 and IPv6 unicast addresses families and instruct the group to be an eBGP session. In this example, any AS that does not match our system-as will be accepted, using the external keyword.

1
2
3


set vrf name VRF1 protocols bgp peer-group CE address-family ipv4-unicast
set vrf name VRF1 protocols bgp peer-group CE address-family ipv6-unicast
set vrf name VRF1 protocols bgp peer-group CE remote-as 'external'

Once the peer-group is defined, we can now add CE1 as a neighbor.

1

set vrf name VRF1 protocols bgp neighbor fd01:101:12::1 peer-group 'CE'

BGP VPN

There are additional parameters we need to configure to identify, export and import customer routes between VRF1 and the global BGP IPv4/IPv6 VPN table.

First is the route distinguisher. It uniquely identifies a route within the global IPv4 and IPv6 VPN tables and allows for address overlapping within multiple VRFs. In this case we are using <ROUTER_ID>:<TABLE_ID> as a scheme. It needs to be configured on both IPv4 and IPv6 address families inside VRF1.

1
2


set vrf name VRF1 protocols bgp address-family ipv4-unicast rd vpn export '10.0.0.2:101'
set vrf name VRF1 protocols bgp address-family ipv6-unicast rd vpn export '10.0.0.2:101'

Next we need to define route targets. These extended communities are used to identify which routes to import from the global IPv4/IPv6 VPN tables and how to mark routes being sent to the global IPv4/IPv6 VPN tables respectively.

1
2
3
4


set vrf name VRF1 protocols bgp address-family ipv4-unicast route-target vpn export '65000:101'
set vrf name VRF1 protocols bgp address-family ipv4-unicast route-target vpn import '65000:101'
set vrf name VRF1 protocols bgp address-family ipv6-unicast route-target vpn export '65000:101'
set vrf name VRF1 protocols bgp address-family ipv6-unicast route-target vpn import '65000:101'

Note

A shortcut option both does exist. Instead of the above the same can be accomplished using:

1
2


set vrf name VRF1 protocols bgp address-family ipv4-unicast route-target vpn both '65000:101'
set vrf name VRF1 protocols bgp address-family ipv6-unicast route-target vpn both '65000:101'

We can now instruct FRR to perform importing and exporting routes to the global IPv4/IPv6 VPN tables:

1
2
3
4


set vrf name VRF1 protocols bgp address-family ipv4-unicast export vpn
set vrf name VRF1 protocols bgp address-family ipv4-unicast import vpn
set vrf name VRF1 protocols bgp address-family ipv6-unicast export vpn
set vrf name VRF1 protocols bgp address-family ipv6-unicast import vpn

Finally, SRv6 is configured for the VRF. Set the SID creation option to per-vrf, creating a unique SID per VRF. The export '101' statement assigns a specific allocation index within the locator prefix for this VRF.

1

set vrf name VRF1 protocols bgp sid vpn per-vrf export '101'

When using a specific index we can predict a resulting SID. In this case 101 is represented as 65 in hexadecimal and produces the SID: 2001:db8:2:aaa:65::/128

Note

Automatic allocation indexing is also an option. Use the following instead of a specific index number:

1

set vrf name VRF1 protocols bgp sid vpn per-vrf export 'auto'

Prior to VyOS 1.5-rolling-202402080022 a reboot is required. This was necessary to ensure that the net.vrf.strict_mode parameter gets set to 1. See the caveat details below.

PE Caveat

Bug

UPDATE: This has been resolved as of VyOS 1.5-rolling-202402080022 via VyOS#2952. The workaround mentioned here is no longer necessary on newer versions.

VyOS PRs #2663 and #2877 attempt to set the kernel parameter net.vrf.strict_mode=1 automatically for you. This parameter is required for SRv6 to function and must be set on both PE routers. After initial configuration a reboot is required in order for this to take effect.

However, after restarting, it does not appear to get correctly set:

1
2


vyos@PE2:~$ sysctl net.vrf.strict_mode
net.vrf.strict_mode = 0

This means FRR is unable to install SRv6 SIDs into the Linux kernel routing table:

1
2
3
4
5
6
7
8
9


vyos@PE2:~$ show ipv6 route bgp
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>r 2001:db8:2:aaa:65::/128 [20/0] is directly connected, VRF1, seg6local End.DT46 table 101, weight 1, 00:40:55

Notice the above prefix shows a r (rejected) status. The following log entries confirm the issue installing into kernel:

1
2
3
4
5


Feb 01 16:58:47 PE2 zebra[1264]: [X5XE1-RS0SW][EC 4043309074] Failed to install Nexthop (14[if 7]) into the kernel
Feb 01 16:58:47 PE2 zebra[1264]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=12, pid=2945720226
Feb 01 16:58:47 PE2 zebra[1264]: [HSYZM-HV7HF] Extended Error: Nexthop id does not exist
Feb 01 16:58:47 PE2 zebra[1264]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: Operation not permitted, type=RTM_NEWNEXTHOP(104), seq=11, pid=2945720226
Feb 01 16:58:47 PE2 zebra[1264]: [HSYZM-HV7HF] Extended Error: Strict mode for VRF is disabled

WORKAROUND: I have observed that manually setting this parameter in the configuration (committing and saving) and rebooting again resolves this issue:

1

set system sysctl parameter net.vrf.strict_mode value '1'

Be sure the other changes are committed before attempting to apply this parameter. Otherwise you may experience a Invalid value error.

After one last reboot, the SID appears to have been programmed correctly:

1
2
3
4
5
6
7
8
9


vyos@PE2:~$ show ipv6 route bgp
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 2001:db8:2:aaa:65::/128 [20/0] is directly connected, VRF1, seg6local End.DT46 table 101, weight 1, 00:01:18

P3

The P router (P3) requires very minimal configuration. The only requirement is to enable IS-IS on both ethernet interfaces and optionally set a few descriptions.

P3 complete configuration

1

set system host-name 'P3'

Interfaces

1
2


set interfaces ethernet eth0 description 'PE2'
set interfaces ethernet eth1 description 'PE4'

IS-IS

1
2
3
4


set protocols isis interface eth0
set protocols isis interface eth1
set protocols isis level 'level-2'
set protocols isis net '49.0000.3333.3333.3333.00'

PE4

PE4 is configured identically to PE2. A complete configuration can be found on GitHub and specific differences are highlighted below below:

1

set system host-name 'PE4'

Interfaces

1

set interfaces loopback lo address '2001:db8:4:ffff::4/128'

Routing

1

set protocols static route6 2001:db8:4::/48 blackhole

IS-IS

1

set protocols isis net '49.0000.4444.4444.4444.00'

Segment Routing

1

set protocols segment-routing srv6 locator L3VPN prefix '2001:db8:4:aaa::/64'

BGP

1
2
3


set protocols bgp parameters router-id '10.0.0.4'
set protocols bgp neighbor 2001:db8:2:ffff::2 description PE2
set protocols bgp neighbor 2001:db8:2:ffff::2 peer-group 'PE'

Customer Configuration

Interfaces

1
2
3
4


set interfaces ethernet eth1 address '172.16.45.4/24'
set interfaces ethernet eth1 address 'fd01:101:2d::4/64'
set interfaces ethernet eth1 description 'CE2'
set interfaces ethernet eth1 vrf 'VRF1'

BGP

1
2


set vrf name VRF1 protocols bgp parameters router-id '10.0.0.4'
set vrf name VRF1 protocols bgp neighbor fd01:101:2d::5 peer-group 'CE'

BGP VPN

1
2


set vrf name VRF1 protocols bgp address-family ipv4-unicast rd vpn export '10.0.0.4:101'
set vrf name VRF1 protocols bgp address-family ipv6-unicast rd vpn export '10.0.0.4:101'

Customer Routers

CE routers only require basic configuration. Substitute addresses and values for each router according to the topology diagram:

1

set system host-name 'CE1'

Interfaces

1
2
3


set interfaces ethernet eth0 description 'PE2'
set interfaces ethernet eth0 address '172.16.12.1/24'
set interfaces ethernet eth0 address 'fd01:101:12::1/64'

BGP

1
2
3
4
5
6


set protocols bgp system-as '65001'
set protocols bgp address-family ipv4-unicast
set protocols bgp address-family ipv6-unicast
set protocols bgp neighbor fd01:101:12::2 address-family ipv4-unicast
set protocols bgp neighbor fd01:101:12::2 address-family ipv6-unicast
set protocols bgp neighbor fd01:101:12::2 remote-as 'external'

Verification

The following can be used to validate the control plane in each PE router.

SRv6 SIDs

Verify a SRv6 End.DT46 SID has been created for VRF1 (table 101):

1
2
3
4
5
6
7
8
9


vyos@PE2:~$ show ipv6 route bgp
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 2001:db8:2:aaa:65::/128 [20/0] is directly connected, VRF1, seg6local End.DT46 table 101, weight 1, 00:00:21

More information about SR Endpoint behaviors can be found in RFC 8986

BGP

IPv4: VPN

Validate routes are in the global IPv4 VPN table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


vyos@PE2:~$ show bgp ipv4 vpn
BGP table version is 1, local router ID is 10.0.0.2, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.2:101
 *> 172.16.12.0/24   0.0.0.0@7<               0         32768 ?
    UN=0.0.0.0 EC{65000:101} label=1616 sid=2001:db8:2:aaa:: sid_structure=[40,24,16,0] type=bgp, subtype=5
Route Distinguisher: 10.0.0.4:101
 *>i172.16.45.0/24   2001:db8:4:ffff::4
                                             0    100      0 ?
    UN=2001:db8:4:ffff::4 EC{65000:101} label=1616 sid=2001:db8:4:aaa:: sid_structure=[40,24,16,0] type=bgp, subtype=0

Displayed  2 routes and 2 total paths

IPv6: VPN

Validate routes are in the global IPv6 VPN table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


vyos@PE2:~$ show bgp ipv6 vpn
BGP table version is 1, local router ID is 10.0.0.2, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.2:101
 *> fd01:101:12::/64 ::@7<                    0         32768 ?
    UN=:: EC{65000:101} label=1616 sid=2001:db8:2:aaa:: sid_structure=[40,24,16,0] type=bgp, subtype=5
Route Distinguisher: 10.0.0.4:101
 *>ifd01:101:2d::/64 2001:db8:4:ffff::4
                                             0    100      0 ?
    UN=2001:db8:4:ffff::4 EC{65000:101} label=1616 sid=2001:db8:4:aaa:: sid_structure=[40,24,16,0] type=bgp, subtype=0

Displayed  2 routes and 2 total paths

IPv4: VRF1

Validate IPv4 routes are received from the CE and are imported into the VRF from other PEs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


vyos@PE2:~$ show bgp vrf VRF1 ipv4
BGP table version is 2, local router ID is 10.0.0.2, vrf id 7
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *> 172.16.12.0/24   0.0.0.0                  0         32768 ?
 *> 172.16.45.0/24   2001:db8:4:ffff::4@0<
                                             0    100      0 ?

Displayed  2 routes and 2 total paths

IPv6: VRF1

Validate IPv6 routes are received from the CE and are imported into the VRF from other PEs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


vyos@PE2:~$ show bgp vrf VRF1 ipv6
BGP table version is 2, local router ID is 10.0.0.2, vrf id 7
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *> fd01:101:12::/64 ::                       0         32768 ?
 *> fd01:101:2d::/64 2001:db8:4:ffff::4@0<
                                             0    100      0 ?

Displayed  2 routes and 2 total paths

Routing Tables

Verify routes are installed into the Linux kernel for the VRF routing table:

IPv4

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


vyos@PE2:~$ show ip route vrf VRF1
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF VRF1:
C>* 172.16.12.0/24 is directly connected, eth1, 00:13:13
B>  172.16.45.0/24 [200/0] via 2001:db8:4:ffff::4 (vrf default) (recursive), label 1616, seg6 2001:db8:4:aaa:65::, weight 1, 00:12:31
  *                          via fe80::5201:ff:fe03:0, eth0 (vrf default), label 1616, seg6 2001:db8:4:aaa:65::, weight 1, 00:12:31

IPv6

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


vyos@PE2:~$ show ipv6 route vrf VRF1
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF VRF1:
C>* fd01:101:12::/64 is directly connected, eth1, 00:13:36
B>  fd01:101:2d::/64 [200/0] via 2001:db8:4:ffff::4 (vrf default) (recursive), label 1616, seg6 2001:db8:4:aaa:65::, weight 1, 00:12:56
  *                            via fe80::5201:ff:fe03:0, eth0 (vrf default), label 1616, seg6 2001:db8:4:aaa:65::, weight 1, 00:12:56
C>* fe80::/64 is directly connected, eth1, 00:13:37

End-to-End connectivity

Caveat

Bug

Before any L3VPN traffic will route properly, traffic must be sent sourced from either PE to the other PE, within the VRF at least once. This must be performed specifically between VRF-member interfaces on PEs.

For example, a ping between PE2’s eth1 interface and PE4’s eth1 interface:

1
2
3
4


vyos@PE2:~$ ping 172.16.45.4 source-address 172.16.12.2 vrf VRF1
PING 172.16.45.5 (172.16.45.5) from 172.16.12.2 : 56(84) bytes of data.
64 bytes from 172.16.45.5: icmp_seq=1 ttl=64 time=1.90 ms
64 bytes from 172.16.45.5: icmp_seq=2 ttl=64 time=1.42 ms

This only needs to be performed once (unless rebooted) and can be completed on either address family. It does not require this action on both families in order for both IPv4 and IPv6 VRF traffic to work. Once completed, all further traffic works as expected.

This is a particularly odd behavior that appears to have something to do with routing within the Linux kernel. Using dropwatch on the inbound PE and collecting dropped packets from CE1 to CE5 (on PE2) I observed the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# PE2

drop at: ip6_pkt_drop+0xeb/0x1c0 (0xffffffff9512123b)
origin: software
input port ifindex: 7
timestamp: Sun Feb  4 19:10:30 2024 028938864 nsec
protocol: 0x86dd
length: 162
original length: 162
drop reason: IP_INNOROUTES

The input port (ifindex: 7) matches the VRF1 interface. The drop reason listed, IP_INNOROUTES, corresponds to a network unreachable error.

These drops are no longer observed after a successful ping between PE2 and PE4’s eth1 interfaces as shown above.

IPv4

Ping from CE1 to CE5

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


vyos@CE1:~$ ping 172.16.45.5
PING 172.16.45.5 (172.16.45.5) 56(84) bytes of data.
64 bytes from 172.16.45.5: icmp_seq=1 ttl=63 time=2.95 ms
64 bytes from 172.16.45.5: icmp_seq=2 ttl=63 time=1.88 ms
64 bytes from 172.16.45.5: icmp_seq=3 ttl=63 time=2.24 ms
64 bytes from 172.16.45.5: icmp_seq=4 ttl=63 time=1.95 ms
64 bytes from 172.16.45.5: icmp_seq=5 ttl=63 time=1.89 ms
^C
--- 172.16.45.5 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 1.878/2.181/2.954/0.407 ms

The packet capture below shows the traffic above collected on PE2’s eth0 interface. This is the interface from PE2 into the provider network facing P3.

Notice the IPv4 traffic is encapsulated into IPv6 with a destination address of the End.DT46 SID allocated for VRF1 (table 101) on PE4.

IPv6

Ping from CE1 to CE5

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


vyos@CE1:~$ ping fd01:101:2d::5
PING fd01:101:2d::5(fd01:101:2d::5) 56 data bytes
64 bytes from fd01:101:2d::5: icmp_seq=1 ttl=63 time=2.68 ms
64 bytes from fd01:101:2d::5: icmp_seq=2 ttl=63 time=1.88 ms
64 bytes from fd01:101:2d::5: icmp_seq=3 ttl=63 time=1.92 ms
64 bytes from fd01:101:2d::5: icmp_seq=4 ttl=63 time=2.13 ms
64 bytes from fd01:101:2d::5: icmp_seq=5 ttl=63 time=2.08 ms
^C
--- fd01:101:2d::5 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 1.878/2.138/2.684/0.289 ms

This packet capture again shows the traffic above collected on PE2, eth0.

Similarly to IPv4 test above, notice the traffic is encapsulated with the same destination address representing the End.DT46 SID allocated for VRF1 (table 101) on PE4.

Conclusion

IPv6 as a transport mechanism for Segment Routing allows for more traditional provider topologies to be simplified significantly as long as there is basic IPv6 connectivity between customer facing routers.

In this lab we created a basic working SRv6 L3VPN network in VyOS. We walked briefly through all of the necessary components and took a peak at how to validate end-to-end connectivity. I hope this was beneficial in providing some background on Segment Routing with IPv6 and how this technology can be used to support Layer 3 VPNs.