In a previous post, I wrote about my experiences with
using FRR on Ubuntu to provide L3VPN services over Segment Routing on IPv6
(SRv6).
Since then a new major version of FRR has become available and I thought it may
be time to revisit this configuration to see if I still experience the
same issues.
Success
TL;DR - I am happy to report that IPv4 and IPv6 L3VPN mostly works out of the
box. However, I found behavioral
caveat.
In this configuration we will configure uSID behaviors and configure IS-IS and
BGP for SRv6.
Configuration
Everything in this post was tested with the following software:
- Debian 12 (Bookworm)
- Linux kernel 5.17.0-1-cloud-amd64
- FRR Version 10.2
Two Linux machines serving as PE1 and PE2 are directly connected over their
ens5 interfaces. This should work without issue as long as there is IS-IS IPv6
reachability between PE devices regardless if intermediary devices are
configured for SRv6.
Linux Kernel
If you are reading this, I assume that you are capable of installing FRR
following their basic instructions found at the top of the page for their
Debian repository.
As noted before these sysctl settings will need to be configured as noted in the
FRR installation instructions.
1
2
3
4
5
6
7
8
9
|
# vi /etc/sysctl.d/local.conf
net.ipv4.conf.all.forwarding=1
net.ipv6.conf.all.forwarding=1
net.ipv6.seg6_flowlabel=1
net.ipv6.conf.all.seg6_enabled=1
## important! ##
net.vrf.strict_mode=1
|
Restart the system or run: sysctl --system to load these settings.
Note
net.vrf.strict_mode=1 is a critical configuration setting. SRv6 routes will
show as rejected (B>r via show ipv6 route in FRR) if this is missing or
skipped. It is important to note that this setting gets reset any time a new VRF
is added to the kernel and must be reset again (including after restarts):
1
|
root@PE1# sysctl net.vrf.strict_mode=1
|
Interfaces
systemd was used configure the following interfaces.
Once these files are in place, reboot the system or:
systemctl reload systemd-networkd.
VRF1
1
2
3
4
5
6
7
8
|
# /etc/systemd/network/0-vrf1.netdev
[NetDev]
Name=VRF1
Kind=vrf
[VRF]
TableId=10
|
1
2
3
4
|
# /etc/systemd/network/0-vrf1.network
[Match]
Name=VRF1
|
Dummy interface: sr0
This interface is
used by isisd
to install SRv6 SIDs in the Linux data plane.
1
2
3
4
|
# /etc/systemd/network/5-sr0.network
[NetDev]
Name=sr0
Kind=dummy
|
1
2
3
|
# /etc/systemd/network/5-sr0.netdev
[Match]
Name=VRF1
|
PE to PE interface: ens4
1
2
3
|
# /etc/systemd/network/10-ens4.network
[NetDev]
Name=ens4
|
1
2
3
|
# /etc/systemd/network/10-ens4.netdev
[Match]
Name=ens4
|
CE to PE interface: ens5
1
2
3
|
# /etc/systemd/network/10-ens5.network
[NetDev]
Name=ens5
|
1
2
3
|
# /etc/systemd/network/10-ens5.netdev
[Match]
Name=ens5
|
FRR
Attributes of the PE1 router:
- Segment Locator Block: fd00:0:1::/48 (name: MAIN)
- Router Loopback: 2001:db8:1::1/128
- SRv6 Interfaces: ens4
- IS-IS Level 2 (49.0001.0000.0000.0001.00)
- L3VPN VRF Interface: ens5 (name: VRF1)
- IPv6 address: 2001:db8:a::1/64
- IPv4 address: 10.0.10.1/24
The configuration is identical on each router with interface addresses, BGP RD
and IS-IS NET changed per router as appropriate.
PE1:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
|
!
frr version 10.2
frr defaults traditional
hostname PE1
service integrated-vtysh-config
!
vrf VRF1
exit-vrf
!
interface ens4
description PE2
ip router isis 1
ipv6 router isis 1
exit
!
interface ens5
description CE1
ip address 10.0.10.1/24
ipv6 address 2001:db8:a::1/64
exit
!
interface lo
ip address 10.0.0.1/32
ip router isis 1
ipv6 address 2001:db8:1:ffff::1/128
ipv6 address fd00:0:1::1/128
ipv6 router isis 1
exit
!
router bgp 65000
bgp router-id 10.0.0.1
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
no bgp network import-check
neighbor PE peer-group
neighbor PE remote-as internal
neighbor PE update-source 2001:db8:1:ffff::1
neighbor PE capability extended-nexthop
neighbor 2001:db8:2:ffff::1 peer-group PE
!
segment-routing srv6
locator MAIN
exit
!
address-family ipv4 vpn
neighbor PE activate
neighbor PE next-hop-self
exit-address-family
!
address-family ipv6 vpn
neighbor PE activate
neighbor PE next-hop-self
exit-address-family
exit
!
router bgp 65000 vrf VRF1
bgp router-id 10.0.0.1
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
no bgp network import-check
sid vpn per-vrf export 101
!
address-family ipv4 unicast
redistribute connected
rd vpn export 10.0.0.1:101
rt vpn both 65000:101
export vpn
import vpn
exit-address-family
!
address-family ipv6 unicast
redistribute connected
rd vpn export 10.0.0.1:101
rt vpn both 65000:101
export vpn
import vpn
exit-address-family
exit
!
router isis 1
net 49.0001.0000.0000.0001.00
segment-routing srv6
locator MAIN
exit
exit
!
segment-routing
srv6
encapsulation
source-address fd00:0:1::1
locators
locator MAIN
prefix fd00:0:1::/48 block-len 32 node-len 16 func-bits 16
behavior usid
format usid-f3216
exit
!
exit
!
formats
format usid-f3216
exit
!
format uncompressed-f4024
exit
!
exit
!
exit
!
exit
!
end
|
PE2:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
|
!
frr version 10.2
frr defaults traditional
hostname PE2
service integrated-vtysh-config
!
vrf VRF1
exit-vrf
!
interface ens4
description PE1
ip router isis 1
ipv6 router isis 1
exit
!
interface ens5
description CE2
ip address 10.0.11.1/24
ipv6 address 2001:db8:b::1/64
exit
!
interface lo
ip address 10.0.0.2/32
ip router isis 1
ipv6 address 2001:db8:2:ffff::1/128
ipv6 address fd00:0:2::1/128
ipv6 router isis 1
exit
!
router bgp 65000
bgp router-id 10.0.0.2
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
no bgp network import-check
neighbor PE peer-group
neighbor PE remote-as internal
neighbor PE update-source 2001:db8:2:ffff::1
neighbor PE capability extended-nexthop
neighbor 2001:db8:1:ffff::1 peer-group PE
!
segment-routing srv6
locator MAIN
exit
!
address-family ipv4 vpn
neighbor PE activate
neighbor PE next-hop-self
exit-address-family
!
address-family ipv6 vpn
neighbor PE activate
neighbor PE next-hop-self
exit-address-family
exit
!
router bgp 65000 vrf VRF1
bgp router-id 10.0.0.2
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
no bgp network import-check
sid vpn per-vrf export 101
!
address-family ipv4 unicast
redistribute connected
rd vpn export 10.0.0.2:101
rt vpn both 65000:101
export vpn
import vpn
exit-address-family
!
address-family ipv6 unicast
redistribute connected
rd vpn export 10.0.0.2:101
rt vpn both 65000:101
export vpn
import vpn
exit-address-family
exit
!
router isis 1
net 49.0001.0000.0000.0002.00
segment-routing srv6
locator MAIN
exit
exit
!
segment-routing
srv6
encapsulation
source-address fd00:0:2::1
locators
locator MAIN
prefix fd00:0:2::/48 block-len 32 node-len 16 func-bits 16
behavior usid
format usid-f3216
exit
!
exit
!
formats
format usid-f3216
exit
!
format uncompressed-f4024
exit
!
exit
!
exit
!
exit
!
end
|
FRR Verification
Within FRR (vtysh), we can verify several attributes about the routing and
SRv6 status:
Verify the global IPv6 routing table:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
PE1# sh ipv6 route
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
K>* ::1/128 [0/256] is directly connected, lo, weight 1, 02:09:59
L * 2001:db8:1:ffff::1/128 is directly connected, lo, weight 1, 02:09:59
C>* 2001:db8:1:ffff::1/128 is directly connected, lo, weight 1, 02:09:59
I>* 2001:db8:2:ffff::1/128 [115/20] via fe80::5201:ff:fe09:1, ens4, weight 1, 00:21:03
I>* fd00:0:1::/48 [115/0] is directly connected, sr0, seg6local End USP, weight 1, 02:09:58
L * fd00:0:1::1/128 is directly connected, lo, weight 1, 02:09:59
C>* fd00:0:1::1/128 is directly connected, lo, weight 1, 02:09:59
B>* fd00:0:1:65::/128 [20/0] is directly connected, VRF1, seg6local End.DT46 table 10, weight 1, 02:09:58
I>* fd00:0:2::/48 [115/10] via fe80::5201:ff:fe09:1, ens4, weight 1, 00:21:03
I>* fd00:0:2::1/128 [115/20] via fe80::5201:ff:fe09:1, ens4, weight 1, 00:21:03
C * fe80::/64 is directly connected, sr0, weight 1, 02:09:59
C * fe80::/64 is directly connected, ens4, weight 1, 02:09:59
C>* fe80::/64 is directly connected, ens3, weight 1, 02:09:59
|
If you notice a rejected route on the End.DT4 segment, ensure that
VRF strict mode
is configured correctly and restart FRR (systemctl restart frr).
uSID Caveat
Bug
You may have noticed the End.DT46 route shown in the above output.
Despite configuring behavior usid on the MAIN segment routing locator, the
programmed behavior was still End.DT46. We were expecting to see the
micro-segment behavior uDT46, however this does not affect traffic flow.
Additionally with uSID behavior in this topology, we were not expecting to see
a Segment Routing Header (SRH) for traffic on the VRF. However, a single-segment
header remained.
IPv4 VPN Verification
Customer device connected to ens5 on PE1:
- IPv4: 10.0.10.2/24
- Gateway: 10.0.10.1
Customer device connected to ens5 on PE2:
- IPv6: 10.0.11.2/24
- Gateway: 10.0.11.1
1
2
3
4
5
6
7
8
|
# Ping CE2 from CE1
> ping 10.0.11.2
84 bytes from 10.0.11.2 icmp_seq=1 ttl=63 time=2.265 ms
84 bytes from 10.0.11.2 icmp_seq=2 ttl=63 time=1.174 ms
84 bytes from 10.0.11.2 icmp_seq=3 ttl=63 time=1.332 ms
84 bytes from 10.0.11.2 icmp_seq=4 ttl=63 time=1.066 ms
84 bytes from 10.0.11.2 icmp_seq=5 ttl=63 time=1.073 ms
|
1
2
3
4
5
6
|
# Traceroute to CE2 to CE1
> trace 10.0.11.2
trace to 10.0.11.2, 8 hops max, press Ctrl+C to stop
1 10.0.0.2 0.892 ms 0.549 ms 0.479 ms
2 *10.0.11.2 0.951 ms (ICMP type:3, code:3, Destination port unreachable)
|
IPv6 VPN Verification
Customer device connected to ens5 on PE1:
- IPv6: 2001:db8:a::2/64
- Gateway: 2001:db8:a::1
Customer device connected to ens5 on PE2:
- IPv6: 2001:db8:b::2/64
- Gateway: 2001:db8:b::1
1
2
3
4
5
6
7
8
|
# Ping CE2 from CE1
> ping 2001:db8:b::2
2001:db8:b::2 icmp6_seq=1 ttl=63 time=2.245 ms
2001:db8:b::2 icmp6_seq=2 ttl=63 time=1.134 ms
2001:db8:b::2 icmp6_seq=3 ttl=63 time=1.232 ms
2001:db8:b::2 icmp6_seq=4 ttl=63 time=1.066 ms
2001:db8:b::2 icmp6_seq=5 ttl=63 time=1.073 ms
|
1
2
3
4
5
6
|
# Traceroute to CE2 to CE1
> trace 2001:db8:b::2
trace to 2001:db8:b::2, 8 hops max, press Ctrl+C to stop
1 2001:db8:1:ffff::1 0.892 ms 0.549 ms 0.479 ms
2 *2001:db8:b::2 0.951 ms (ICMP type:3, code:3, Destination port unreachable)
|
Summary
It appears the L3VPN issues I previously have had trouble with have mostly been
resolved. Micro-segment SIDs still seem to be questionable (barring mistakes
caused by myself).
I look forward to more experimentation and enabling this functionality
in VyOS.