Segment Routing, in short called SR too, has been implemented in a range of network operating systems. First successful working implementation happened in year 2014. I have been curious about the state of SR in FRR since some time but did not have the right router appliance that would be easy enough to setup and easy to maintain. Now this is possible, the blog article is over here. Building a small FRRouting router in less than 10 minutes.

The FRR implementation lacks at the moment lots of functionality, many things are not working so do not await anything fancy from it. But it is good enough for a networking lab to see how things are supposed to work, and what how things are actually implemented in FRR router daemon.

The SR implementation in FRR is a demo tech presentation using the OSPF process. It shows it has been put work and first effort to get segment routing running in FRRouting daemon.

📘 Note
This is an experimental implementation of segment routing in FRR. It was never meant for use in production networks.

In a previous blog entry a setup routine is explained, how to create a small FRR virtual router appliance for networking labs. This particular FRR appliance is used in explained examples.

Following steps explained below are necessary be applied to the router appliance additionally to get segment routing working. SR works in FRR only for IPv4. SR relies on the MPLS linux kernel modules to be available and loaded in the kernel.

Prerequisites

The setup steps explained below are need to get SR working. SR will use the MPLS data plane to distribute label information. MPLS linux modules are needed for this operation and a correct setting for interfaces to be used, and the maximal amount of available labels.

Use a current linux version in the Router appliance. Linux version used in this lab:

user % uname -a Linux frr 5.4.8-0-virt #1-Alpine SMP Tue, 07 Jan 2020 12:27:21 UTC i686 Linux

It is recommended to use the most recent FRR routing version, here it is version 7.2:

user % vtysh

Hello, this is FRRouting (version 7.2). Copyright 1996-2005 Kunihiro Ishiguro, et al.

Load the necessary linux kernel MPLS modules:

modprobe mpls_router
modprobe mpls_gso
modprobe mpls_iptunnel

Verify kernel modules are loaded using the lsmod | grep mpls command:

root # lsmod | grep mpls

mpls_iptunnel 16384 0 mpls_gso 16384 0 mpls_router 28672 1 mpls_iptunnel ip_tunnel 24576 1 mpls_router

Append these 3 kernel modules to the /etc/modules-load.d/modules.conf configuration file. This ensures modules are loaded on each system start.

📄 File /etc/modules-load.d/modules.conf

mpls_router mpls_gso mpls_iptunnel

Activate MPLS for all 5 interfaces in the FRR appliance, and set the maximal available platform MPLS label stack to 1048575. Label with the label ID=1048575 will be the last possible label for this platform.

sysctl -w net.mpls.conf.eth0.input=1
sysctl -w net.mpls.conf.eth1.input=1
sysctl -w net.mpls.conf.eth2.input=1
sysctl -w net.mpls.conf.eth3.input=1
sysctl -w net.mpls.conf.lo.input=1
sysctl -w net.mpls.platform_labels=1048575

For permanent setting across reboots, create following file /etc/sysctl.d/91-mpls.conf

📄 File /etc/sysctl.d/91-mpls.conf

net.mpls.conf.lo.input=1 net.mpls.conf.eth0.input=1 net.mpls.conf.eth1.input=1 net.mpls.conf.eth2.input=1 net.mpls.conf.eth3.input=1 net.mpls.platform_labels=1048575

The FRR router appliance is now prepared to be used as a template for the SR networking lab. It is safe to poweroff the appliance. Create a new router template in GNS3 with 4 ethernet interfaces using this special setup appliance as explained in steps above.

Network topology setup

3 routers. Full mesh network topology. All routers are directly connected.

FRR SR OSPF network topology with IP adressing:

       lo                                          lo
    192.0.2.1                                   192.0.2.2
    +-------+                                   +-------+
    |       | eth0                         eth0 |       |
    | FRR1  |-----------------------------------| FRR2  |
    |       | .1/30                       .2/30 |       |
    +-------+                                   +-------+
   eth2 | .10/30                              .5/30 | eth1
        |                                           |
        |                                           |
        |                                           |
        |                                           |
        |                                           |
        |                                           |
        |                                           |
        |                 +-------+                 |
        |           .9/30 |       | .6/30           |
        +-----------------| FRR3  |-----------------+
                     eth2 |       | eth1
                          +-------+
                             lo
                          192.0.2.3

Setup this topology in the network simulator application.

Router configuration

Current FRR SR implementation uses the OSPF SPF algorithm to calculate the network topology. The configuration consists of 2 parts. * 1st part - Configure OSPF on routers to get IP reachability in the network topology * 2nd part - Enable specific FRR OSPF features to enable and configure SR

OSPF process

  • Create OSPF router process and assingn a router-id. The router-id is identical with the IP of the loopback lo interface.
  • Create loopback address and announce its network to area 0.
  • Configure IP point-to-point links between network routers use (/30) transfer networks, and announce prefixes to area 0

Initial router configuration for FRR1:

conf t
!
router ospf
 ospf router-id 192.0.2.1
exit
!
interface lo
 ip address 192.0.2.1/32
 ip ospf area 0
exit
!
interface eth0
 ip address 10.0.0.1/30
 ip ospf area 0
 ip ospf network point-to-point
 no shut
exit
!
interface eth1
 ip address 10.0.0.10/30
 ip ospf area 0
 ip ospf network point-to-point
 no shut
end
!

Configuration steps seen above need to be adjusted to match correct settings for router FRR2 and router FRR3. The initial OSPF configuration for all 3 FRR networking lab routers is available below:

After configuration, verify if the loopback IP addresses of FRR2 192.0.2.2 and FRR3 192.0.2.3 are in the routing table. The last 2 entries:

FRR1# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

O   10.0.0.0/30 [110/100] is directly connected, eth0, 00:00:59
C>* 10.0.0.0/30 is directly connected, eth0, 01:06:36
O>* 10.0.0.4/30 [110/200] via 10.0.0.2, eth0, 00:00:42
  *                       via 10.0.0.9, eth2, 00:00:42
O   10.0.0.8/30 [110/100] is directly connected, eth2, 00:00:45
C>* 10.0.0.8/30 is directly connected, eth2, 01:06:36
O   192.0.2.1/32 [110/0] is directly connected, lo, 00:00:42
C>* 192.0.2.1/32 is directly connected, lo, 01:06:36
O>* 192.0.2.2/32 [110/100] via 10.0.0.2, eth0, 00:00:45
O>* 192.0.2.3/32 [110/100] via 10.0.0.9, eth2, 00:00:44

Routers FRR2 and FRR3 loopack IP address are in the routing table. This demonstrates functioning IP reachability in network.

SR Segment Routing

The SR uses the OSPF Opaque LSA Option for routing information flooding within the backbone area.

The FRR OSPF router process needs to have the capability opaque command configured. SR uses 3 different OPAQUE LSA's in OSPF to carry the information about:

  • Router Information:: flood the SR capabilities of the node.
    • node-msd and
    • segment-routing global-block 16000 23999
    • supported algorithm, the only available and the default is SPF
  • Extended Link: flood the Adjaceny and Lan Adjacency Segment Identifier
  • Extended Prefix: flood the Prefix Segment Identifier

Here in example the default SRGB Segment Routing Global Block is used. This is the block 16000 - 23999. A great presentation from Clarence Filsfils about the SRGB can found on the official segment routing website.

To enable flooding of the LSA's to the OSPF area use the router-info area command is used. This configuration example is for the router FRR1, identified by the router-address setting.

conf t
!
router ospf
 capability opaque
  mpls-te on
  mpls-te router-address 192.0.2.1
 router-info area
 segment-routing on
 segment-routing global-block 16000 23999
 segment-routing node-msd 8
 segment-routing prefix 192.0.2.1/32 index 1021
end

FRR2 and FRR3 configuration should be adjusted to match the network topology setup. All router fully working configurations are available below:

Verification

Following commands are used to show the routing protocol information:

sh ip route

Show the converged IP network:

FRR1# sh ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

O   10.0.0.0/30 [110/100] is directly connected, eth0, 01:37:50
C>* 10.0.0.0/30 is directly connected, eth0, 01:37:51
O>* 10.0.0.4/30 [110/200] via 10.0.0.2, eth0, 01:35:28
  *                       via 10.0.0.9, eth2, 01:35:28
O   10.0.0.8/30 [110/100] is directly connected, eth2, 01:37:50
C>* 10.0.0.8/30 is directly connected, eth2, 01:37:51
O   192.0.2.1/32 [110/0] is directly connected, lo, 01:37:50
C>* 192.0.2.1/32 is directly connected, lo, 01:37:51
O>* 192.0.2.2/32 [110/100] via 10.0.0.2, eth0, 01:35:38
O>* 192.0.2.3/32 [110/100] via 10.0.0.9, eth2, 01:35:28

show mpls table

Display MPLS label distribution, and take a look at the Inbound Label ID:

FRR1# show mpls table
 Inbound                            Outbound
   Label     Type          Nexthop     Label
--------  -------  ---------------  --------
   17002       SR         10.0.0.2  implicit-null
   50004       SR         10.0.0.9  implicit-null
   50005       SR         10.0.0.9  implicit-null
   50006       SR         10.0.0.2  implicit-null
   50007       SR         10.0.0.2  implicit-null

show ip ospf

Show the running OSPF process and verify the OpaqCapability flag is set to enabled, notice the output of opaque area LSA 12 in the last line of the output:

FRR1# sh ip ospf
 OSPF Routing Process, Router ID: 192.0.2.1
 Supports only single TOS (TOS0) routes
 This implementation conforms to RFC2328
 RFC1583Compatibility flag is disabled
 OpaqueCapability flag is enabled
 Initial SPF scheduling delay 0 millisec(s)
 Minimum hold time between consecutive SPFs 50 millisec(s)
 Maximum hold time between consecutive SPFs 5000 millisec(s)
 Hold time multiplier is currently 2
 SPF algorithm last executed 25m45s ago
 Last SPF duration 763 usecs
 SPF timer is inactive
 LSA minimum interval 5000 msecs
 LSA minimum arrival 1000 msecs
 Write Multiplier set to 20
 Refresh timer 10 secs
 Number of external LSA 0. Checksum Sum 0x00000000
 Number of opaque AS LSA 0. Checksum Sum 0x00000000
 Number of areas attached to this router: 1
 Area ID: 0.0.0.0 (Backbone)
   Number of interfaces in this area: Total: 3, Active: 3
   Number of fully adjacent neighbors in this area: 2
   Area has no authentication
   SPF algorithm executed 9 times
   Number of LSA 15
   Number of router LSA 3. Checksum Sum 0x0001ca89
   Number of network LSA 0. Checksum Sum 0x00000000
   Number of summary LSA 0. Checksum Sum 0x00000000
   Number of ASBR summary LSA 0. Checksum Sum 0x00000000
   Number of NSSA LSA 0. Checksum Sum 0x00000000
   Number of opaque link LSA 0. Checksum Sum 0x00000000
   Number of opaque area LSA 12. Checksum Sum 0x00067bc2

show ip ospf database

Check the OSPF database. It shows 3 router link states, having 5 links each. Notify the Area-Local Opaque-LSA section displayed:

    FRR1# sh ip ospf database

           OSPF Router with ID (192.0.2.1)

            Router Link States (Area 0.0.0.0)

    Link ID         ADV Router      Age  Seq#       CkSum  Link count
    192.0.2.1       192.0.2.1        679 0x80000009 0x5919 5
    192.0.2.2       192.0.2.2        681 0x80000009 0x7305 5
    192.0.2.3       192.0.2.3        683 0x80000005 0xfe6b 5

                    Area-Local Opaque-LSA (Area 0.0.0.0)

    Opaque-Type/Id  ADV Router      Age  Seq#       CkSum
    4.0.0.0         192.0.2.1        674 0x80000001 0xba06
    4.0.0.0         192.0.2.2        733 0x80000001 0xb40b
    4.0.0.0         192.0.2.3        733 0x80000001 0xae10
    7.0.0.1         192.0.2.1        682 0x80000001 0x6490
    7.0.0.1         192.0.2.2        733 0x80000001 0x866b
    7.0.0.1         192.0.2.3        733 0x80000001 0xa846
    8.0.0.1         192.0.2.3        733 0x80000001 0x65a6
    8.0.0.2         192.0.2.3        678 0x80000001 0x0bf4
    8.0.0.5         192.0.2.1        674 0x80000001 0xe429
    8.0.0.5         192.0.2.2        675 0x80000001 0xb05d
    8.0.0.6         192.0.2.1        682 0x80000001 0x5ea1
    8.0.0.6         192.0.2.2        733 0x80000001 0x679f

show ip ospf database segment-routing

It took me some time to find the command to verify SR operation. Following commands are currently available:

FRR1# sh ip ospf database segment-routing

CR Carriage Return adv-router Advertising SR node json JavaScript Object Notation self-originate Self-originated SR node

Display self originated IP prefixes. Notify the SRGB size of 8000 labels and its first label 16000.

FRR1# show ip ospf database segment-routing self-originate

    OSPF Segment Routing database for ID 192.0.2.1

SR-Node: 192.0.2.1      SRGB (Size/Label): 8000/16000   Algorithm(s): SPF       MSD: 8

    Prefix or Link  Label In  Label Out       Node or Adj. SID  Interface          Nexthop
------------------  --------  ---------  ---------------------  ---------  ---------------
      192.0.2.1/32         0          0         SR Pfx (idx 1)         lo        192.0.2.1
      10.0.0.10/32     50005        pop    SR Adj. (lbl 50005)       eth2         10.0.0.9
      10.0.0.10/32     50004        pop    SR Adj. (lbl 50004)       eth2         10.0.0.9
       10.0.0.1/32     50007        pop    SR Adj. (lbl 50007)       eth0         10.0.0.2
       10.0.0.1/32     50006        pop    SR Adj. (lbl 50006)       eth0         10.0.0.2

Show all prefixes in the OSPF SR DB of FRR1. Verify the MSD is set to 8, notice the algorithm(s) preset SPF:

FRR1# sh ip ospf database segment-routing

            OSPF Segment Routing database for ID 192.0.2.1

SR-Node: 192.0.2.1      SRGB (Size/Label): 8000/16000   Algorithm(s): SPF       MSD: 8

    Prefix or Link  Label In  Label Out       Node or Adj. SID  Interface          Nexthop
------------------  --------  ---------  ---------------------  ---------  ---------------
      192.0.2.1/32         0          0         SR Pfx (idx 1)         lo        192.0.2.1
      10.0.0.10/32     50005        pop    SR Adj. (lbl 50005)       eth2         10.0.0.9
      10.0.0.10/32     50004        pop    SR Adj. (lbl 50004)       eth2         10.0.0.9
       10.0.0.1/32     50007        pop    SR Adj. (lbl 50007)       eth0         10.0.0.2
       10.0.0.1/32     50006        pop    SR Adj. (lbl 50006)       eth0         10.0.0.2

SR-Node: 192.0.2.2      SRGB (Size/Label): 8000/16000   Algorithm(s): SPF       MSD: 8

    Prefix or Link  Label In  Label Out       Node or Adj. SID  Interface          Nexthop
------------------  --------  ---------  ---------------------  ---------  ---------------
      192.0.2.2/32     17022        pop         SR Pfx (idx 2)       eth0         10.0.0.2

SR-Node: 192.0.2.3      SRGB (Size/Label): 8000/16000   Algorithm(s): SPF       MSD: 8

    Prefix or Link  Label In  Label Out       Node or Adj. SID  Interface          Nexthop
------------------  --------  ---------  ---------------------  ---------  ---------------
      192.0.2.3/32     17023        pop         SR Pfx (idx 3)       eth2         10.0.0.9

The MPLS label distribution is done using SR segment routing. LDP is not used. While using IPv4 with SR then SR is the control plane and MPLS is the data plane.

For the IPv6 SR the data plane is IPv6. Segment routing support in the linux kernel is already working verification using the dmesg command:

user % dmesg | grep -i segment [ 7.753507] Segment Routing with IPv6

At this moment FRR lacks the control plane like SR working with IPv6. The FRR ospf6 routing daemon has not the needed functionality in the code:

FRR1(config)# router ospf6
FRR1(config-ospf6)# ?
  area                   OSPF6 area parameters
  auto-cost              Calculate OSPF interface cost according to bandwidth
  distance               Administrative distance
  end                    End current mode and change to enable mode
  exit                   Exit current mode and down to previous mode
  find                   Find CLI command matching a regular expression
  interface              Enable routing on an IPv6 interface
  list                   Print command list
  log-adjacency-changes  Log changes in adjacency state
  no                     Negate a command or set its defaults
  ospf6                  Open Shortest Path First (OSPF) for IPv6
  output                 Direct vtysh output to file
  quit                   Exit current mode and down to previous mode
  redistribute           Redistribute
  stub-router            Make router a stub router
  timers                 Adjust routing timers

Just out of curiosity isisd routing protocol lacks SR too:

FRR1(config)# router isis 1
FRR1(config-router)# ?
  area-password          Configure the authentication password for an area
  default-information    Control distribution of default information
  domain-password        Set the authentication password for a routing domain
  end                    End current mode and change to enable mode
  exit                   Exit current mode and down to previous mode
  find                   Find CLI command matching a regular expression
  hostname               Dynamic hostname for IS-IS
  is-type                IS Level for this routing process (OSI only)
  list                   Print command list
  log-adjacency-changes  Log changes in adjacency state
  lsp-gen-interval       Minimum interval between regenerating same LSP
  lsp-mtu                Configure the maximum size of generated LSPs
  lsp-refresh-interval   LSP refresh interval
  max-lsp-lifetime       Maximum LSP lifetime
  metric-style           Use old-style (ISO 10589) or new-style packet formats
  mpls-te                MPLS-TE specific commands
  net                    A Network Entity Title for this process (OSI only)
  no                     Negate a command or set its defaults
  output                 Direct vtysh output to file
  purge-originator       Use the RFC 6232 purge-originator
  quit                   Exit current mode and down to previous mode
  redistribute           Redistribute information from another routing protocol
  set-attached-bit       Set attached bit to identify as L1/L2 router for inter-area traffic
  set-overload-bit       Set overload bit to avoid any transit traffic
  spf-delay-ietf         IETF SPF delay algorithm
  spf-interval           Minimum interval between SPF calculations
  topology               Configure IS-IS topologie

The SR for isisd might prove easier to implement:

FRR3(config-router)# topology ?
  ipv4-mgmt       IPv4 management topology
  ipv4-multicast  IPv4 multicast topology
  ipv4-unicast    IPv4 unicast topology
  ipv6-dstsrc     IPv6 dst-src topology
  ipv6-mgmt       IPv6 management topology
  ipv6-multicast  IPv6 multicast topology
  ipv6-unicast    IPv6 unicast topology

OSPF is written on top of IP. This is the reason why FRR and many other popular network operating system implementations, have a separate OSPF router processes for each IP address family. FRR has ospf for IP and ospf6 for IPv6.

While implementing a solution for isisd. Solving the problem once and for both address families, IP and IPv6. The control plane the IS-IS routing protocol does not depend on the data plane, here IP and IPv6.

It would be a great project to code the SRv6 support for FRR. An opportunity to learn more about segment routing, IPv6, and its ospf, ospf6 and isis routing daemons. Learn about linux routing to gain programming experience while implementing the solution.

Someone has to make a first step to get things going.

References