This is a follow up netlab article about EXOS ESRP simple setup configuration. The prerequisite is the EXOS ESRP simple setup. In this blog entry 2 new OSPF are added to the network topology, RS1 and RS2, OSPF configuration is added to all routers in the network topology. RS5 is only a switch, nothing is added there. OSPF configuration is added to verify how this vendor described ESRP design is behaving having fully working network topology. This is the simplest EXOS ESRP setup scenario taken from of the Extreme Networks official documentation.
Once again I want to mention this is a blind configuration, I have no experience at all with ESRP, just being curious to see this networking example and learn about EXOS ESRP configuration and operation and how quickly it converges using this example.
network topology
OSPF configuration is added to all topology routers, on top of already running and configured ESRP article from previous blog entry.
| sysName | ospf router-id |
|---|---|
| RS1 | 1.1.1.1 |
| RS2 | 2.2.2.2 |
| RS3 | 5.5.5.5 |
| RS4 | 5.5.5.5 |
Network topology showing relevant IP and OSPF settings:
1.1.1.1 2.2.2.2
+-------+ +-------+
| | 12 12 | |
| RS1 |-----------------------------------| RS2 |
| | | |
+-------+ +-------+
10 | | 10
| (OSPF) |
| |
| |
5.5.5.5 5.5.5.5
+-------+ +-------+
| | 3 3 | |
| RS3 +-----------------------------------+ RS4 |
| | | |
+---+---+ +---+---+
1 | | 1
| ESRP VR1 |
| 10.1.2.3 |
| |
| +-------+ |
| | | |
+-----------------+ RS5 +-----------------+
| |
+-+---+-+
| |
+-------------+ +-------------+
| eth0 eth0 |
+---+---+ +---+---+
| | | |
| node1 | | node2 |
| | | |
+-------+ +-------+
Added port numbers to the network topology above to have a overview how routers are connected in case someone wants to test it by yourself.
EXOS version
EXOS version used in this netlab:
Card Partition Installation Date Version Name Branch ------------------------------------------------------------------------------ Switch primary Thu Jan 16 13:06:22 UTC 2025 32.7.2.19 rescue.xos 32.7.2.19 Switch secondary Thu Jan 16 13:06:22 UTC 2025 32.7.2.19 rescue.xos 32.7.2.19
Configuration
Configuration of loopback interfaces on EXOS was not familiar, adding a loopback interface to EXOS is done in following way:
- create vlan named
lo0 - configure IP addressing
- enable
loopback modeon interface
For RS1/RS2 the full configuration is exaplained below. RS3 and RS3 has added OSPF only to establish OSPF neighours. RS3 and RS4 have no loopback interfaces since it does not anything useful having loopback addresses on RS3 and RS4, it would only clutter the example.
RS1
Configuration steps for setting up RS1 router:
- Remove ports 10 and 12 from the Default VLAN
- Create vlan vlan12 with a 802.1Q tag of 12
- Create vlan vlan13 with a 802.1Q tag of 13
- Configure VLAN 12 with an IP address
- Configure VLAN 13 with an IP address
- Add port 12 to VLAN 12 untagged
- Add port 10 to VLAN 13 untagged
- Enable ipforwarding for VLAN vlan10 and vlan13
- create int lo0 vlan, without VLAN tag
- Enable ipforwarding for VLAN lo0
- configure OSPF
Configure commands:
#
# RS1
#
configure snmp sysName RS1
#
configure vlan default delete port 10,12
create vlan vlan12 tag 12
create vlan vlan13 tag 13
configure vlan vlan12 ipaddress 10.0.0.0/31
configure vlan vlan13 ipaddress 10.0.0.2/31
configure vlan vlan12 add port 12 untagged
configure vlan vlan13 add port 10 untagged
enable ipforwarding vlan vlan12
enable ipforwarding vlan vlan13
#
create vlan lo0
configure vlan lo0 ipaddress 1.1.1.1/32
enable ipforwarding vlan lo0
enable loopback-mode vlan lo0
#
configure ospf routerid 1.1.1.1
configure ospf add vlan vlan12 area 0.0.0.0 link-type point-to-point
configure ospf add vlan vlan13 area 0.0.0.0 link-type point-to-point
configure ospf add vlan lo0 area 0.0.0.0 passive
enable ospf
#
save
y
RS2
Same configuration commands, as for RS1 only wit
#
# RS2
#
configure snmp sysname RS2
#
configure vlan default delete port 10,12
create vlan vlan12 tag 12
create vlan vlan24 tag 24
configure vlan vlan12 ipaddress 10.0.0.1/31
configure vlan vlan24 ipaddress 10.0.0.4/31
configure vlan vlan12 add port 12 untagged
configure vlan vlan24 add port 10 untagged
enable ipforwarding vlan vlan12
enable ipforwarding vlan vlan24
#
create vlan lo0
configure vlan lo0 ipaddress 2.2.2.2/32
enable ipforwarding vlan lo0
enable loopback-mode vlan lo0
#
configure ospf routerid 2.2.2.2
configure ospf add vlan vlan12 area 0.0.0.0 link-type point-to-point
configure ospf add vlan vlan24 area 0.0.0.0 link-type point-to-point
configure ospf add vlan lo0 area 0.0.0.0 passive
enable ospf
#
save
y
RS3
Adding OSPF configuration to ESRP router R3. R3 and R4 have identical OSPF router-id set here 5.5.5.5. No loopback interface is added here.
Adding OSPF configuration to ESRP router R3:
- Remove ports 10 from the Default VLAN
- Create VLAN TAG 13
- Configure VLAN 13 IP addressing
- Add port 10 to vlan13 untagged
- Enable ipforwarding on vlan13
Configuration commands:
#
# RS3
#
configure vlan default delete port 10
create vlan vlan13 tag 13
configure vlan vlan13 ipaddress 10.0.0.3/31
configure vlan vlan13 add port 10 untagged
enable ipforwarding vlan vlan13
#
disable ospf
configure ospf routerid 5.5.5.5
configure ospf add vlan vlan13 area 0.0.0.0 link-type point-to-point
enable ospf
#
save
y
RS4
Configuration commands for R4:
#
# RS4
#
configure vlan default delete port 10
create vlan vlan24 tag 24
configure vlan vlan24 ipaddress 10.0.0.5/31
configure vlan vlan24 add port 10 untagged
enable ipforwarding vlan vlan24
#
disable ospf
configure ospf routerid 5.5.5.5
configure ospf add vlan vlan24 area 0.0.0.0 link-type point-to-point
enable ospf
#
save
y
node1
Using 2 linux nodes in this example for testing, following IP addressing is used:
Configuration commands
ip addr add 10.1.2.11/24 dev eth0
ip route add default via 10.1.2.3
node2
Configuration commands:
ip addr add 10.1.2.12/24 dev eth0
ip route add default via 10.1.2.3
Full router configurations
The R100 - 10.255.255.100 - (FRRouting) is connected to port1 on router RS2, using SVI/vlan named vlan255:
- RS1 full configuration
- RS2 full configuration
- RS3 full configuration
- RS4 full configuration
- R100 FRR full configuration
Verify OSPF
When the configuration is finished, and the routing protocol converged, inspect the OSPF state using following commands:
show ospf neighbor
Show the ip routing table:
show iproute
Verify routing table output of RS1 and RS2 and check the times shown how long IP prefixes are available in the routing table. Verify the times of the advertised prefix 10.1.2.0/24:
The 10.255.255.100 is a directly connected OSPF FRR appliance, this FRR router will be used further for the routing table inspection and the events that it receives. FRR is not depicted in the network topology above, it is directly connected to RS2, and it has only a monitoring function.
RS1
OSPF neighbours:
RS1.4 # show ospf neighbor
Neighbor ID Pri State Up/Dead Time Address Interface
BFD Session State
==========================================================================================
2.2.2.2 1 FULL /DROTHER 00:01:17:12/00:00:00:02 10.0.0.1 vlan12
None
5.5.5.5 1 FULL /DROTHER 00:01:20:56/00:00:00:06 10.0.0.3 vlan13
None
Total number of neighbors: 2 (All neighbors in Full state)
The routing table of router RS1
RS1.8 # show iproute
Ori Destination Gateway Mtr Flags VLAN Duration
#d 1.1.1.1/32 1.1.1.1 1 U------um--f- lo0 0d:1h:27m:55s
#oa 2.2.2.2/32 10.0.0.1 15 UG-D---um--f- vlan12 0d:1h:18m:16s
#d 10.0.0.0/31 10.0.0.0 1 U------um--f- vlan12 0d:1h:28m:18s
#d 10.0.0.2/31 10.0.0.2 1 U------um--f- vlan13 0d:1h:28m:14s
#oa 10.0.0.4/31 10.0.0.1 10 UG-D---um--f- vlan12 0d:1h:18m:16s
#oa 10.0.0.6/31 10.0.0.1 10 UG-D---um--f- vlan12 0d:1h:18m:16s
#oa 10.1.2.0/24 10.0.0.3 10 UG-D---um--f- vlan13 0d:0h:0m:3s
#oa 10.255.255.100/32 10.0.0.1 10 UG-D---um--f- vlan12 0d:1h:18m:4s
RS2
OSPF neighbor state, check the 10.255.255.100 is connected to SVI named vlan255.
RS2.2 # show ospf neighbor
Neighbor ID Pri State Up/Dead Time Address Interface
BFD Session State
==========================================================================================
1.1.1.1 1 FULL /DROTHER 00:01:20:05/00:00:00:02 10.0.0.0 vlan12
None
5.5.5.5 1 FULL /DROTHER 00:01:20:05/00:00:00:05 10.0.0.5 vlan24
None
10.255.255.100 1 FULL /DROTHER 00:01:19:58/00:00:00:08 10.0.0.7 vlan255
None
Total number of neighbors: 3 (All neighbors in Full state)
IP routing table of RS2:
RS2.3 # show iproute
Ori Destination Gateway Mtr Flags VLAN Duration
#oa 1.1.1.1/32 10.0.0.0 15 UG-D---um--f- vlan12 0d:1h:20m:21s
#d 2.2.2.2/32 2.2.2.2 1 U------um--f- lo0 0d:1h:28m:23s
#d 10.0.0.0/31 10.0.0.1 1 U------um--f- vlan12 0d:1h:28m:29s
#oa 10.0.0.2/31 10.0.0.0 10 UG-D---um--f- vlan12 0d:1h:20m:21s
#d 10.0.0.4/31 10.0.0.4 1 U------um--f- vlan24 0d:1h:28m:29s
#d 10.0.0.6/31 10.0.0.6 1 U------um--f- vlan255 0d:1h:20m:40s
#oa 10.1.2.0/24 10.0.0.0 15 UG-D---um--f- vlan12 0d:0h:0m:4s
#oa 10.255.255.100/32 10.0.0.7 5 UG-D---um--f- vlan255 0d:1h:20m:5s
RS3
OSPF neighbour output, this router has only 1 OSPF neighbour, R1:
RS3.4 # show ospf neighbor
Neighbor ID Pri State Up/Dead Time Address Interface
BFD Session State
==========================================================================================
1.1.1.1 1 FULL /DROTHER 00:01:27:34/00:00:00:07 10.0.0.2 vlan13
None
Total number of neighbors: 1 (All neighbors in Full state)
The routing table of RS3, verify that hash # attached to the prefix 10.1.2.0/24
RS3.5 # show iproute
Ori Destination Gateway Mtr Flags VLAN Duration
#oa 1.1.1.1/32 10.0.0.2 15 UG-D---um--f- vlan13 0d:1h:24m:43s
#oa 2.2.2.2/32 10.0.0.2 20 UG-D---um--f- vlan13 0d:1h:20m:58s
#oa 10.0.0.0/31 10.0.0.2 10 UG-D---um--f- vlan13 0d:1h:24m:43s
#d 10.0.0.2/31 10.0.0.3 1 U------um--f- vlan13 0d:1h:27m:41s
#oa 10.0.0.4/31 10.0.0.2 15 UG-D---um--f- vlan13 0d:1h:20m:58s
#oa 10.0.0.6/31 10.0.0.2 15 UG-D---um--f- vlan13 0d:1h:20m:58s
#d 10.1.2.0/24 10.1.2.3 1 U------um--f- sales 0d:1h:27m:41s
#oa 10.255.255.100/32 10.0.0.2 15 UG-D---um--f- vlan13 0d:1h:20m:45s
The hash # indicates this is currently the ESRP ACTIVE router.
(#) Preferred unicast and multicast route.
RS4
OSPF neighbour output, this router has also only 1 OSPF neighbour, R2:
RS4.1 # show ospf neighbor
Neighbor ID Pri State Up/Dead Time Address Interface
BFD Session State
==========================================================================================
2.2.2.2 1 FULL /DROTHER 00:03:09:23/00:00:00:02 10.0.0.4 vlan24
None
Total number of neighbors: 1 (All neighbors in Full state)
Notice that here the # is lacking in the output:
RS4.2 # show iproute
Ori Destination Gateway Mtr Flags VLAN Duration
#oa 1.1.1.1/32 10.0.0.4 20 UG-D---um--f- vlan24 0d:3h:9m:27s
#oa 2.2.2.2/32 10.0.0.4 15 UG-D---um--f- vlan24 0d:3h:9m:27s
#oa 10.0.0.0/31 10.0.0.4 10 UG-D---um--f- vlan24 0d:3h:9m:27s
#oa 10.0.0.2/31 10.0.0.4 15 UG-D---um--f- vlan24 0d:3h:9m:27s
#d 10.0.0.4/31 10.0.0.5 1 U------um--f- vlan24 0d:3h:9m:41s
#oa 10.0.0.6/31 10.0.0.4 10 UG-D---um--f- vlan24 0d:3h:9m:27s
d 10.1.2.0/24 10.1.2.3 1 -------um---- sales 0d:3h:9m:41s
#oa 10.255.255.100/32 10.0.0.4 10 UG-D---um--f- vlan24 0d:3h:9m:27s
RS4 is here the ESRP Slave router, it is lacking the # attached to the highly available prefix. RS4 is the ESRP PASSIVE router.
Result
Testing when all routers are working on a converged network. Nothing is being shut down or powered off. Just all components running all the time, stable state. Here the results from the linux nodes pinging the RS1 and RS2 routers.
node1
This is the reachability for one of the backbone routers 1.1.1.1 while ESRP is converged and stable:
... --- 1.1.1.1 ping statistics --- 100 packets transmitted, 61 received, 39% packet loss, time 100151ms rtt min/avg/max/mdev = 0.656/2.541/3.913/1.113 ms
Big packet loss. 39% packets lost out of 100 sent
node2
This is the reachabilty of one of the backbone routers 2.2.2.2:
... --- 2.2.2.2 ping statistics --- 100 packets transmitted, 19 received, 81% packet loss, time 101074ms rtt min/avg/max/mdev = 1.059/1.886/3.911/0.972 ms
Big packet loss, targets are not reachable most of the time on a stable network.
RS1
Command output on RS1, showing the same, the IP prefix is oscillating, flapping:
RS1.2 # show iproute 10.1.2.0/24 | grep number.of.routes
Total number of routes = 0
RS1.2 # show iproute 10.1.2.0/24 | grep number.of.routes
Total number of routes = 1
RS1.2 # show iproute 10.1.2.0/24 | grep number.of.routes
Total number of routes = 0
RS1.2 # show iproute 10.1.2.0/24 | grep number.of.routes
Total number of routes = 1
In the command output below you can observe how long the prefix is available in the routing table, not longer than 5 seconds, then it disappears again:
RS1.14 # show iproute | grep "#oa 10.1.2.0/24"
RS1.14 # show iproute | grep "#oa 10.1.2.0/24"
#oa 10.1.2.0/24 10.0.0.3 10 UG-D---um--f- vlan13 0d:0h:0m:0s
RS1.14 # show iproute | grep "#oa 10.1.2.0/24"
#oa 10.1.2.0/24 10.0.0.3 10 UG-D---um--f- vlan13 0d:0h:0m:3s
R1.14 # show iproute | grep "#oa 10.1.2.0/24"
It is available ~5 seconds, then removed from the routing table.
RS2
The prefix is frequently added and removed from the routing tables, it is flapping all of the time. Output from RS2 showing exactly the same:
RS2.4 # show iproute 10.1.2.0/24 | grep number.of.routes
Total number of routes = 0
RS2.5 # show iproute 10.1.2.0/24 | grep number.of.routes
Total number of routes = 1
RS2.5 # show iproute 10.1.2.0/24 | grep number.of.routes
Total number of routes = 0
RS2.5 # show iproute 10.1.2.0/24 | grep number.of.routes
Total number of routes = 1
FRR events
I have added a third FRR appliance and configured as one of OSPF backbone routers. This is the OSPF router having the router-id 10.255.255.100.
The Using the ip monitor linux CLI command verify the events to the 10.1.2.0/24 IP prefix:
Deleted 10.1.2.0/24 nhid 12 via 10.0.0.6 dev eth0 proto ospf metric 20 10.1.2.0/24 nhid 12 via 10.0.0.6 dev eth0 proto ospf metric 20 Deleted 10.1.2.0/24 nhid 12 via 10.0.0.6 dev eth0 proto ospf metric 20 10.1.2.0/24 nhid 12 via 10.0.0.6 dev eth0 proto ospf metric 20 Deleted 10.1.2.0/24 nhid 12 via 10.0.0.6 dev eth0 proto ospf metric 20 10.1.2.0/24 nhid 12 via 10.0.0.6 dev eth0 proto ospf metric 20 Deleted 10.1.2.0/24 nhid 12 via 10.0.0.6 dev eth0 proto ospf metric 20
ICMP test from RS1
Now sending ICMP packets to a node in the 10.1.2.0/24 network from router RS1
Ping(ICMP) 10.1.2.11: 4 packets, 8 data bytes, interval 1 second(s). 16 bytes from 10.1.2.11: icmp_seq=0 ttl=63 time=3.938 ms Packet transmit error; Destination unreachable Packet transmit error; Destination unreachable Packet transmit error; Destination unreachable --- 10.1.2.11 ping statistics --- 1 packets transmitted, 1 packets received, 0% loss round-trip min/avg/max = 3/3/3 ms
Output stats do not display packet loss, the target is not reachable, the router reports Packet trasmit error. While sending 5 packets only 1 packet successfully transmitted and only 1 received. Other ICMP packets have not reached the IP target dues to transmit error.
Summary
The IP prefix 10.1.2.0/24 is oscillating in the backbone every ~5 seconds when adding OSPF on top of the example network topology. And that only to using identical OSPF routerid on both ESRP routers. RS3 and RS4. The OSPF backbone area should to be stable all of the time. Not to mention that is only the IP view, layer 3. I did not look deep enough what is happening with Spanning Tree in this example, most probably nothing, but this add another level of complexity to the example. It gets even more ridiculous when adding more prefixes on both ESRP routers.
The ESRP topology above will need some correcting steps and I know I have NOT have missed one single step in this configuration. The official documentation does not mention anything about the OSPF state when ESRP is active, when you have to configure the same OSPF routerid 5.5.5.5, and why it is a important step in this network topology. The reasons are unknown to me.
I have notified the EXOS support on their documentation website but as a not a paying customer chances are none they actually reply. Setting OSPF router-id to identical value generates a lot of link advertise events when and makes it failing. Especially when many links are connected this way, you sure do not want to operate this in your production network.
In a follow up blog entry I will try to make this ESPR design work by correcting few settings, adding a new protocol and removing links. I am not sure how it is gonna work out, but it is worth a try out.