# Segment Routing IPv6 (SRv6) with FRR and Debian


In a previous [post](/2022/06/27/srv6-frr), I wrote about my experiences with
using FRR on Ubuntu to provide L3VPN services over Segment Routing on IPv6
(SRv6).

Since then a new major version of FRR has become available and I thought it may
be time to revisit this configuration to see if I still experience the
[same issues](/2022/06/27/srv6-frr/#ipv4-l3vpn-issues). 

{{< admonition type=success >}}
TL;DR - I am happy to report that IPv4 and IPv6 L3VPN mostly works out of the
box. However, I found behavioral [caveat](#usid-caveat).
{{< /admonition >}}

In this configuration we will configure uSID behaviors and configure IS-IS and
BGP for SRv6.

## Configuration

Everything in this post was tested with the following software:
* Debian 12 (Bookworm)
* Linux kernel 5.17.0-1-cloud-amd64
* FRR Version 10.2

Two Linux machines serving as `PE1` and `PE2` are directly connected over their
`ens4` interfaces. This should work without issue as long as there is IS-IS IPv6
reachability between PE devices regardless if intermediary devices are
configured for SRv6.

Customer edge devices are connected (`ens5`) to the same L3VPN service (VRF) on
each side.

### Linux Kernel

If you are reading this, I assume that you are capable of installing FRR
following their basic instructions found at the top of the page for their 
[Debian repository](https://deb.frrouting.org/).

As noted before these sysctl settings will need to be configured as noted in the 
[FRR installation instructions](http://docs.frrouting.org/en/latest/installation.html).

```
# vi /etc/sysctl.d/local.conf

net.ipv4.conf.all.forwarding=1
net.ipv6.conf.all.forwarding=1
net.ipv6.seg6_flowlabel=1
net.ipv6.conf.all.seg6_enabled=1

## important! ##
net.vrf.strict_mode=1
```

Restart the system or run: `sysctl --system` to load these settings.

{{< admonition type=note >}}
`net.vrf.strict_mode=1` is a critical configuration setting. SRv6 routes will
show as rejected (`B>r` via `show ipv6 route` in FRR) if this is missing or
skipped. It is important to note that this setting gets reset any time a new VRF
is added to the kernel and must be reset again (including after restarts):

```
root@PE1# sysctl net.vrf.strict_mode=1
```
{{< /admonition >}}

### Interfaces

[systemd](https://systemd.network) was used configure the following interfaces.
Once these files are in place, reboot the system or:
`systemctl reload systemd-networkd`.

#### VRF1
```
# /etc/systemd/network/0-vrf1.netdev

[NetDev]
Name=VRF1
Kind=vrf

[VRF]
TableId=10
```

```
# /etc/systemd/network/0-vrf1.network

[Match]
Name=VRF1
```

#### Dummy interface: sr0

This interface is
[used by isisd](https://docs.frrouting.org/en/stable-10.2/isisd.html#clicmd-interface-NAME)
to install SRv6 SIDs in the Linux data plane.

```
# /etc/systemd/network/5-sr0.netdev
[NetDev]
Name=sr0
Kind=dummy
```

```
# /etc/systemd/network/5-sr0.network
[Match]
Name=VRF1
```

#### PE to PE interface: ens4

```
# /etc/systemd/network/10-ens4.netdev
[NetDev]
Name=ens4
```

```
# /etc/systemd/network/10-ens4.network
[Match]
Name=ens4
```

#### CE to PE interface: ens5

```
# /etc/systemd/network/10-ens5.netdev
[NetDev]
Name=ens5
```

```
# /etc/systemd/network/10-ens5.network
[Match]
Name=ens5
```

### FRR

Attributes of the PE1 router:

* Segment Locator Block: fd00:0:1::/48   (name: MAIN)
* Router Loopback: 2001:db8:1::1/128
* SRv6 Interfaces: ens4
    * IS-IS Level 2 (49.0001.0000.0000.0001.00)
* L3VPN VRF Interface: ens5 (name: VRF1)
    * IPv6 address: 2001:db8:a::1/64
    * IPv4 address: 10.0.10.1/24

The configuration is identical on each router with interface addresses, BGP RD
and IS-IS NET changed per router as appropriate.

PE1:
```
!
frr version 10.2
frr defaults traditional
hostname PE1
service integrated-vtysh-config
!
vrf VRF1
exit-vrf
!
interface ens4
 description PE2
 ip router isis 1
 ipv6 router isis 1
exit
!
interface ens5
 description CE1
 ip address 10.0.10.1/24
 ipv6 address 2001:db8:a::1/64
exit
!
interface lo
 ip address 10.0.0.1/32
 ip router isis 1
 ipv6 address 2001:db8:1:ffff::1/128
 ipv6 address fd00:0:1::1/128
 ipv6 router isis 1
exit
!
router bgp 65000
 bgp router-id 10.0.0.1
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 neighbor PE peer-group
 neighbor PE remote-as internal
 neighbor PE update-source 2001:db8:1:ffff::1
 neighbor PE capability extended-nexthop
 neighbor 2001:db8:2:ffff::1 peer-group PE
 !
 segment-routing srv6
  locator MAIN
 exit
 !
 address-family ipv4 vpn
  neighbor PE activate
  neighbor PE next-hop-self
 exit-address-family
 !
 address-family ipv6 vpn
  neighbor PE activate
  neighbor PE next-hop-self
 exit-address-family
exit
!
router bgp 65000 vrf VRF1
 bgp router-id 10.0.0.1
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 sid vpn per-vrf export 101
 !
 address-family ipv4 unicast
  redistribute connected
  rd vpn export 10.0.0.1:101
  rt vpn both 65000:101
  export vpn
  import vpn
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
  rd vpn export 10.0.0.1:101
  rt vpn both 65000:101
  export vpn
  import vpn
 exit-address-family
exit
!
router isis 1
 net 49.0001.0000.0000.0001.00
 segment-routing srv6
  locator MAIN
 exit
exit
!
segment-routing
 srv6
  encapsulation
   source-address fd00:0:1::1
  locators
   locator MAIN
    prefix fd00:0:1::/48 block-len 32 node-len 16 func-bits 16
    behavior usid
    format usid-f3216
   exit
   !
  exit
  !
  formats
   format usid-f3216
   exit
   !
   format uncompressed-f4024
   exit
   !
  exit
  !
 exit
 !
exit
!
end
```

PE2:
```
!
frr version 10.2
frr defaults traditional
hostname PE2
service integrated-vtysh-config
!
vrf VRF1
exit-vrf
!
interface ens4
 description PE1
 ip router isis 1
 ipv6 router isis 1
exit
!
interface ens5
 description CE2
 ip address 10.0.11.1/24
 ipv6 address 2001:db8:b::1/64
exit
!
interface lo
 ip address 10.0.0.2/32
 ip router isis 1
 ipv6 address 2001:db8:2:ffff::1/128
 ipv6 address fd00:0:2::1/128
 ipv6 router isis 1
exit
!
router bgp 65000
 bgp router-id 10.0.0.2
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 neighbor PE peer-group
 neighbor PE remote-as internal
 neighbor PE update-source 2001:db8:2:ffff::1
 neighbor PE capability extended-nexthop
 neighbor 2001:db8:1:ffff::1 peer-group PE
 !
 segment-routing srv6
  locator MAIN
 exit
 !
 address-family ipv4 vpn
  neighbor PE activate
  neighbor PE next-hop-self
 exit-address-family
 !
 address-family ipv6 vpn
  neighbor PE activate
  neighbor PE next-hop-self
 exit-address-family
exit
!
router bgp 65000 vrf VRF1
 bgp router-id 10.0.0.2
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 no bgp network import-check
 sid vpn per-vrf export 101
 !
 address-family ipv4 unicast
  redistribute connected
  rd vpn export 10.0.0.2:101
  rt vpn both 65000:101
  export vpn
  import vpn
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected
  rd vpn export 10.0.0.2:101
  rt vpn both 65000:101
  export vpn
  import vpn
 exit-address-family
exit
!
router isis 1
 net 49.0001.0000.0000.0002.00
 segment-routing srv6
  locator MAIN
 exit
exit
!
segment-routing
 srv6
  encapsulation
   source-address fd00:0:2::1
  locators
   locator MAIN
    prefix fd00:0:2::/48 block-len 32 node-len 16 func-bits 16
    behavior usid
    format usid-f3216
   exit
   !
  exit
  !
  formats
   format usid-f3216
   exit
   !
   format uncompressed-f4024
   exit
   !
  exit
  !
 exit
 !
exit
!
end
```

### FRR Verification

Within FRR (`vtysh`), we can verify several attributes about the routing and
SRv6 status:

Verify the global IPv6 routing table:
```
PE1# sh ipv6 route
Codes: K - kernel route, C - connected, L - local, S - static,
       R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* ::1/128 [0/256] is directly connected, lo, weight 1, 02:09:59
L * 2001:db8:1:ffff::1/128 is directly connected, lo, weight 1, 02:09:59
C>* 2001:db8:1:ffff::1/128 is directly connected, lo, weight 1, 02:09:59
I>* 2001:db8:2:ffff::1/128 [115/20] via fe80::5201:ff:fe09:1, ens4, weight 1, 00:21:03
I>* fd00:0:1::/48 [115/0] is directly connected, sr0, seg6local End USP, weight 1, 02:09:58
L * fd00:0:1::1/128 is directly connected, lo, weight 1, 02:09:59
C>* fd00:0:1::1/128 is directly connected, lo, weight 1, 02:09:59
B>* fd00:0:1:65::/128 [20/0] is directly connected, VRF1, seg6local End.DT46 table 10, weight 1, 02:09:58
I>* fd00:0:2::/48 [115/10] via fe80::5201:ff:fe09:1, ens4, weight 1, 00:21:03
I>* fd00:0:2::1/128 [115/20] via fe80::5201:ff:fe09:1, ens4, weight 1, 00:21:03
C * fe80::/64 is directly connected, sr0, weight 1, 02:09:59
C * fe80::/64 is directly connected, ens4, weight 1, 02:09:59
C>* fe80::/64 is directly connected, ens3, weight 1, 02:09:59
```

If you notice a rejected route on the End.DT4 segment, ensure that 
[VRF strict mode](#linux-kernel) 
is configured correctly and restart FRR (`systemctl restart frr`).

#### uSID Caveat

{{< admonition type=bug >}}
You may have noticed the `End.DT46` route shown in the above output.

Despite configuring `behavior usid` on the `MAIN` segment routing locator, the
programmed behavior was still `End.DT46`. We were expecting to see the
micro-segment behavior `uDT46`, however this does not affect traffic flow.

Additionally with uSID behavior in this topology, we were not expecting to see
a Segment Routing Header (SRH) for traffic on the VRF. However, a single-segment
header remained.
{{< /admonition >}}

### IPv4 VPN Verification

Customer device connected to `ens5` on PE1:
* IPv4: 10.0.10.2/24
* Gateway: 10.0.10.1

Customer device connected to `ens5` on PE2:
* IPv6: 10.0.11.2/24
* Gateway: 10.0.11.1

```
# Ping CE2 from CE1
> ping 10.0.11.2

84 bytes from 10.0.11.2 icmp_seq=1 ttl=63 time=2.265 ms
84 bytes from 10.0.11.2 icmp_seq=2 ttl=63 time=1.174 ms
84 bytes from 10.0.11.2 icmp_seq=3 ttl=63 time=1.332 ms
84 bytes from 10.0.11.2 icmp_seq=4 ttl=63 time=1.066 ms
84 bytes from 10.0.11.2 icmp_seq=5 ttl=63 time=1.073 ms
```

```
# Traceroute to CE2 to CE1
> trace 10.0.11.2

trace to 10.0.11.2, 8 hops max, press Ctrl+C to stop
 1   10.0.0.2   0.892 ms  0.549 ms  0.479 ms
 2   *10.0.11.2   0.951 ms (ICMP type:3, code:3, Destination port unreachable)
```

### IPv6 VPN Verification

Customer device connected to `ens5` on PE1:
* IPv6: 2001:db8:a::2/64
* Gateway: 2001:db8:a::1

Customer device connected to `ens5` on PE2:
* IPv6: 2001:db8:b::2/64
* Gateway: 2001:db8:b::1

```
# Ping CE2 from CE1
> ping 2001:db8:b::2

2001:db8:b::2 icmp6_seq=1 ttl=63 time=2.245 ms
2001:db8:b::2 icmp6_seq=2 ttl=63 time=1.134 ms
2001:db8:b::2 icmp6_seq=3 ttl=63 time=1.232 ms
2001:db8:b::2 icmp6_seq=4 ttl=63 time=1.066 ms
2001:db8:b::2 icmp6_seq=5 ttl=63 time=1.073 ms
```

```
# Traceroute to CE2 to CE1
> trace 2001:db8:b::2

trace to 2001:db8:b::2, 8 hops max, press Ctrl+C to stop
 1   2001:db8:1:ffff::1   0.892 ms  0.549 ms  0.479 ms
 2   *2001:db8:b::2   0.951 ms (ICMP type:3, code:3, Destination port unreachable)
```

## Summary

It appears the L3VPN issues I previously have had trouble with have mostly been
resolved. Micro-segment SIDs still seem to be questionable (barring mistakes
caused by myself).

I look forward to more experimentation and enabling this functionality
in VyOS.