Policy Based Routing and Default Gateway disappearing

Hi all,

I have a problem that has me baffled.

I have several Ubuntu 18.04.02 LTS Virtual machines that all use policy based routing. The are all on the same networks same 2 internet connections etc.

The problem I have is that randomly the default gateway disappears. The PBR tables are still there but the gateway goes bye bye. So incoming traffic is ok but default outgoing traffic dies.

Here are some details

root@WWW-1:/# ip rule list
0: from all lookup local
32762: from 10.11.11.204 lookup FW1
32763: from 10.10.11.204 lookup LAN1
32764: from 10.11.10.204 lookup FW2
32765: from 10.10.10.204 lookup LAN2
32766: from all lookup main
32767: from all lookup default

root@WWW-1:/# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 ens160
10.10.11.0 0.0.0.0 255.255.255.0 U 0 0 0 ens192
10.11.10.0 0.0.0.0 255.255.255.0 U 0 0 0 ens224
10.11.11.0 0.0.0.0 255.255.255.0 U 0 0 0 ens256

Script in crontab to build PBR on bootup

@reboot root /etc/scripts/routing

Script

#!/bin/bash

ip route add 10.10.10.0/24 dev ens160 src 10.10.10.204 table LAN2
ip route add default via 10.10.10.1 table LAN2

ip route add 10.11.10.0/24 dev ens224 src 10.11.10.204 table FW2
ip route add default via 10.11.10.1 table FW2

ip route add 10.10.11.0/24 dev ens192 src 10.10.11.204 table LAN1
ip route add default via 10.10.11.3 table LAN1

ip route add 10.11.11.0/24 dev ens256 src 10.11.11.204 table FW1
ip route add default via 10.11.11.3 table FW1

ip route add 10.10.10.0/24 dev ens160 src 10.10.10.204
ip route add 10.11.10.0/24 dev ens224 src 10.11.10.204

ip route add 10.10.11.0/24 dev ens192 src 10.10.11.204
ip route add 10.11.11.0/24 dev ens256 src 10.11.11.204

ip rule add from 10.10.10.204 table LAN2
ip rule add from 10.11.10.204 table FW2

ip rule add from 10.10.11.204 table LAN1
ip rule add from 10.11.11.204 table FW1

ip route add default via 10.11.10.1

Lastly - There is no default gateway set in the netplan config.

If I reboot or if I put the gateway back in manually everything looks ok..

WWW-1:/# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.11.10.1 0.0.0.0 UG 0 0 0 ens224
10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 ens160
10.10.11.0 0.0.0.0 255.255.255.0 U 0 0 0 ens192
10.11.10.0 0.0.0.0 255.255.255.0 U 0 0 0 ens224
10.11.11.0 0.0.0.0 255.255.255.0 U 0 0 0 ens256

Does anyone have any ideas why this would happen ? sometimes it's good for months sometimes not.