# Revisiting Segment Routing IPv6 (SRv6) with VyOS At the beginning of 2024, I looked at configuring a very basic SRv6 L3VPN service using VyOS. During that effort, I ran into a [critical caveat](/2024/02/01/srv6-vyos/#caveat) in which CE traffic was not being forwarded until locally sourced traffic on each PE was transmitted. ## Issue The trace below demonstrates a sequence of CE1 sourced ICMP echo packets destined for CE5. We can see that they were encapsulated in an SRv6 packet by noting the destination prefix as the `End.DT46` SID on PE4 (`2001:db8:4:aaa:65::`): ``` # tracer: nop # # entries-in-buffer/entries-written: 2743/2743 #P:1 # # _-----=> irqs-off/BH-disabled # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | | -0 [000] b.s1. 3833.771767: fib6_table_lookup: table 101 oif 0 iif 7 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113 -0 [000] b.s1. 3833.782525: fib6_table_lookup: table 101 oif 0 iif 7 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113 -0 [000] b.s1. 3833.793467: fib6_table_lookup: table 101 oif 0 iif 7 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113 -0 [000] b.s1. 3833.804406: fib6_table_lookup: table 101 oif 0 iif 7 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113 ``` Each of these packet's routing process attempts a lookup on `table 101` (VRF1). However, no route to `2001:db8:4:aaa:65::` exists within the VRF table. The result is the behavior I noted in a [previous post](/2024/02/01/srv6-vyos/#caveat). The packet is silently dropped at this stage with a [IP_INNOROUTES](https://github.com/torvalds/linux/blob/d6ef8b40d075c425f548002d2f35ae3f06e9cf96/include/net/dropreason-core.h#L377) error: `network unreachable` indicating no route to the destination can be found. {{< admonition type=note >}} Unfortunately VyOS does not include the appropriate tracing options out of the box. The above traces were obtained by compiling a custom kernel with the appropriate tracing options enabled. Once running, the following trace options were performed: ``` root@PE2:~# echo 1 > /sys/kernel/tracing/events/fib6/fib6_table_lookup/enable root@PE2:~# echo 1 > /sys/kernel/tracing/tracing_on root@PE2:~# ping 172.16.45.5 vrf VRF1 count 10 root@PE2:~# echo 0 > /sys/kernel/tracing/tracing_on root@PE2:~# cat /sys/kernel/debug/tracing/trace ``` {{< /admonition >}} This behavior was not something I experienced when attempting a [similar configuration](/2024/12/16/srv6-frr/) on a vanilla Debian installation. Here is the result of that trace performing the same lookup: ``` # tracer: nop # # entries-in-buffer/entries-written: 31/31 #P:1 # # _-----=> irqs-off/BH-disabled # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | | -0 [000] b.s1. 8907.437537: fib6_table_lookup: table 255 oif 0 iif 6 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113 -0 [000] b.s1. 8907.437543: fib6_table_lookup: table 10 oif 0 iif 6 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113 -0 [000] b.s1. 8907.437546: fib6_table_lookup: table 254 oif 0 iif 6 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev ens3 gw fe80::5201:ff:fe09:0 err 0 ``` This highlighted a difference between the route lookup behavior between VyOS and Linux in general. Linux utilizes a form of [policy based routing](https://www.man7.org/linux/man-pages/man8/ip-rule.8.html) (`ip rules`) to determine the priority of which tables to consult in a first match exit strategy. From a generic Debian installation out of the box, this policy appears as: ``` debian@PE2:~$ ip -6 rule show 0: from all lookup local 1000: from all lookup [l3mdev-table] 32766: from all lookup main ``` On VyOS there is a difference most notably after `l3mdev-table` (VRF) lookups: ``` vyos@PE2:~$ ip -6 rule show 1000: from all lookup [l3mdev-table] 2000: from all lookup [l3mdev-table] unreachable 32765: from all lookup local 32766: from all lookup main 32767: from all lookup default ``` VyOS does [modify](https://github.com/vyos/vyos-1x/blob/77d5b096bb91964b7abc9fc495f574333184edbf/src/conf_mode/vrf.py#L274) this behavior on the creation of VRFs. When the lookup is performed, no route is found at priority `1000`. The very next result at `2000` drops the packet with an unreachable error if no match was found previously. I suspect this rule serves to prevent VRF isolated traffic from errantly resolving a next-hop in another table as doing so would allow traffic to "leak" out of a contained environment. However in our case, this statement is preventing SRv6 encapsulated L3VPN traffic from being able to recurse the `main` table in order to forward traffic appropriately. In my opinion, the SRv6 encapsulation and routing behavior should not be in conflict with these priorities. I suspect this discovered routing behavior should be classified as a bug in `seg6` within the kernel itself. The VRF route appears as: ``` vyos@vyos:~$ ip route show 172.16.45.0/24 vrf VRF1 172.16.45.0/24 nhid 31 encap seg6 mode encap segs 1 [ 2001:db8:4:aaa:65:: ] via inet6 fe80::5201:ff:fe04:0 dev eth0 proto bgp metric 20 ``` This route has all of the elements to correctly identify the SR instructions, next-hop, and outbound interface. I believe that the resulting SRv6 packet should simply follow the predetermined routing table forwarding instruction: `via inet6 fe80::5201:ff:fe04:0 dev eth0`. ## Workaround In order to work around this behavior we need to modify the route lookup policy. Fortunately, VyOS includes [local route policy](https://docs.vyos.io/en/latest/configuration/policy/local-route.html) as a standard configuration feature. Here we can simply short circuit the routing behavior to perform a lookup differently under specific circumstances. A working policy in this case needs to: 1. Match the SRv6 source address of the PE. 2. Match the SRv6 destination prefix. 3. Instruct the lookup to be performed on the `main` table. 4. Match the above conditions before following the normal lookup process. We can use the following configuration to achieve this: ### PE2 ``` policy { local-route6 { rule 500 { destination { address "2001:db8:4::/48" } set { table "main" } source { address "2001:db8:2:ffff::2" } } } } ``` ### PE4 ``` policy { local-route6 { rule 500 { destination { address "2001:db8:2::/48" } set { table "main" } source { address "2001:db8:4:ffff::4" } } } } ``` ### Result The above configuration modifies the `ip -6 rule` table as: ``` vyos@PE2:~$ ip -6 rule show 500: from 2001:db8:2:ffff::2 to 2001:db8:4::/48 lookup main 1000: from all lookup [l3mdev-table] 2000: from all lookup [l3mdev-table] unreachable 32765: from all lookup local 32766: from all lookup main ``` In order of priority, the rules are matched and route caching is performed. Our CE traffic within the VRF is encapsulated in SRv6 and the resulting lookup will be from `2001:db8:2:ffff::2` with an IPv6 destination within: `2001:db8:4::/48`. It will now match the first rule (priority `500`) and be forced to perform a lookup using the `main` table instead of the VRF's. It is important to note that I very specifically match the SRv6 encapsulation source address (loopback) to avoid introducing any other undesired traffic flow leaking across tables. Any other transit SRv6 traffic is already in the `main` table and not effected by this change.