Home Posts AS1050

Contents

Revisiting Segment Routing IPv6 (SRv6) with VyOS

Jonathan Voss included in Networking
   1228 words   6 minutes 

At the beginning of 2024, I looked at configuring a very basic SRv6 L3VPN service using VyOS. During that effort, I ran into a critical caveat in which CE traffic was not being forwarded until locally sourced traffic on each PE was transmitted.

Issue

The trace below demonstrates a sequence of CE1 sourced ICMP echo packets destined for CE5. We can see that they were encapsulated in an SRv6 packet by noting the destination prefix as the End.DT46 SID on PE4 (2001:db8:4:aaa:65::):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# tracer: nop
#
# entries-in-buffer/entries-written: 2743/2743   #P:1
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
          <idle>-0       [000] b.s1.  3833.771767: fib6_table_lookup: table 101 oif 0 iif 7 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113
          <idle>-0       [000] b.s1.  3833.782525: fib6_table_lookup: table 101 oif 0 iif 7 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113
          <idle>-0       [000] b.s1.  3833.793467: fib6_table_lookup: table 101 oif 0 iif 7 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113
          <idle>-0       [000] b.s1.  3833.804406: fib6_table_lookup: table 101 oif 0 iif 7 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113

Each of these packet’s routing process attempts a lookup on table 101 (VRF1). However, no route to 2001:db8:4:aaa:65:: exists within the VRF table. The result is the behavior I noted in a previous post. The packet is silently dropped at this stage with a IP_INNOROUTES error: network unreachable indicating no route to the destination can be found.

Note

Unfortunately VyOS does not include the appropriate tracing options out of the box. The above traces were obtained by compiling a custom kernel with the appropriate tracing options enabled.

Once running, the following trace options were performed:

1
2
3
4
5

root@PE2:~# echo 1 > /sys/kernel/tracing/events/fib6/fib6_table_lookup/enable
root@PE2:~# echo 1 > /sys/kernel/tracing/tracing_on
root@PE2:~# ping 172.16.45.5 vrf VRF1 count 10
root@PE2:~# echo 0 > /sys/kernel/tracing/tracing_on
root@PE2:~# cat /sys/kernel/debug/tracing/trace

This behavior was not something I experienced when attempting a similar configuration on a vanilla Debian installation. Here is the result of that trace performing the same lookup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# tracer: nop
#
# entries-in-buffer/entries-written: 31/31   #P:1
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
          <idle>-0       [000] b.s1.  8907.437537: fib6_table_lookup: table 255 oif 0 iif 6 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113
          <idle>-0       [000] b.s1.  8907.437543: fib6_table_lookup: table  10 oif 0 iif 6 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev lo gw :: err -113
          <idle>-0       [000] b.s1.  8907.437546: fib6_table_lookup: table 254 oif 0 iif 6 proto 43 2001:db8:2:ffff::2/0 -> 2001:db8:4:aaa:65::/0 tos 0 scope 0 flags 0 ==> dev ens3 gw fe80::5201:ff:fe09:0 err 0

This highlighted a difference between the route lookup behavior between VyOS and Linux in general. Linux utilizes a form of policy based routing (ip rules) to determine the priority of which tables to consult in a first match exit strategy.

From a generic Debian installation out of the box, this policy appears as:

1
2
3
4

debian@PE2:~$ ip -6 rule show
0:	from all lookup local
1000:	from all lookup [l3mdev-table]
32766:	from all lookup main

On VyOS there is a difference most notably after l3mdev-table (VRF) lookups:

1
2
3
4
5
6

vyos@PE2:~$ ip -6 rule show
1000:	from all lookup [l3mdev-table]
2000:	from all lookup [l3mdev-table] unreachable
32765:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

VyOS does modify this behavior on the creation of VRFs. When the lookup is performed, no route is found at priority 1000. The very next result at 2000 drops the packet with an unreachable error if no match was found previously. I suspect this rule serves to prevent VRF isolated traffic from errantly resolving a next-hop in another table as doing so would allow traffic to “leak” out of a contained environment.

However in our case, this statement is preventing SRv6 encapsulated L3VPN traffic from being able to recurse the main table in order to forward traffic appropriately.

In my opinion, the SRv6 encapsulation and routing behavior should not be in conflict with these priorities. I suspect this discovered routing behavior should be classified as a bug in seg6 within the kernel itself.

The VRF route appears as:

1
2

vyos@vyos:~$ ip route show 172.16.45.0/24 vrf VRF1
172.16.45.0/24 nhid 31  encap seg6 mode encap segs 1 [ 2001:db8:4:aaa:65:: ] via inet6 fe80::5201:ff:fe04:0 dev eth0 proto bgp metric 20

This route has all of the elements to correctly identify the SR instructions, next-hop, and outbound interface. I believe that the resulting SRv6 packet should simply follow the predetermined routing table forwarding instruction: via inet6 fe80::5201:ff:fe04:0 dev eth0.

Workaround

In order to work around this behavior we need to modify the route lookup policy. Fortunately, VyOS includes local route policy as a standard configuration feature.

Here we can simply short circuit the routing behavior to perform a lookup differently under specific circumstances. A working policy in this case needs to:

  1. Match the SRv6 source address of the PE.
  2. Match the SRv6 destination prefix.
  3. Instruct the lookup to be performed on the main table.
  4. Match the above conditions before following the normal lookup process.

We can use the following configuration to achieve this:

PE2

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

policy {
    local-route6 {
        rule 500 {
            destination {
                address "2001:db8:4::/48"
            }
            set {
                table "main"
            }
            source {
                address "2001:db8:2:ffff::2"
            }
        }
    }
}

PE4

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

policy {
    local-route6 {
        rule 500 {
            destination {
                address "2001:db8:2::/48"
            }
            set {
                table "main"
            }
            source {
                address "2001:db8:4:ffff::4"
            }
        }
    }
}

Result

The above configuration modifies the ip -6 rule table as:

1
2
3
4
5
6

vyos@PE2:~$ ip -6 rule show
500:	from 2001:db8:2:ffff::2 to 2001:db8:4::/48 lookup main
1000:	from all lookup [l3mdev-table]
2000:	from all lookup [l3mdev-table] unreachable
32765:	from all lookup local
32766:	from all lookup main

In order of priority, the rules are matched and route caching is performed. Our CE traffic within the VRF is encapsulated in SRv6 and the resulting lookup will be from 2001:db8:2:ffff::2 with an IPv6 destination within: 2001:db8:4::/48. It will now match the first rule (priority 500) and be forced to perform a lookup using the main table instead of the VRF’s.

It is important to note that I very specifically match the SRv6 encapsulation source address (loopback) to avoid introducing any other undesired traffic flow leaking across tables. Any other transit SRv6 traffic is already in the main table and not effected by this change.

Updated on 26 Dec 2024
Read Markdown
 Segment Routing, IPv6, VyOS
Back | Home