SBC HA Cluster: Instability of the ethernet driver r8169

[JLSeagull]

I was getting crazy with High Availability Cluster just after updating of the nodes from Debian 9 to 10 and I've spent some days to discover the reason. As troubleshooting of error of cluster is a bit complex I'd like to share the solution with the Community.

After the upgrading the SBC HA Cluster had worked fine for some time and suddenly both nodes has stopped of see each other online, both were up, able to communicate with each other and connected to the PBX, though. I was receiving notifications from the cluster minute to minute. Analyzing the behaviour of ping between node 1 and node 2 and among the nodes and the PBX, I perceived that the loss rate of packs was very very high in both nodes. The reason: Debian Buster had updated the NIC' driver for a version incredibly unstable (r8169) during the upgrading of the Operating System. Both nodes were using the driver r8168 before the upgrading and working fine. Debian Buster had updated the driver for r8169 and things got crazy. To fix this error I had to downgrading the driver from r8169 to r8168.

Below a summary of my nodes with the configuration that is working fine again.

SBC1:

NIC: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 10); driver=r8168 driverversion=8.046.00
OS: debian 10.12 (x86-64) - Kernel: 4.9.0-16-amd64


SBC2:

NIC: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 02); driver=r8168 driverversion=8.046.00
OS: debian 10.12 (x86-64) - Kernel: 4.19.0-21-amd64

Link relating the problem with the driver r8169:

https://www.linuxquestions.org/questions/linux-hardware-18/realtek-rtl8111-8168-8411-ethernet-controller-r8168-driver-install-r8169-driver-doesn't-work-4175641982/

 

[SteveITS]

Thanks for posting the solution. A couple random notes...

3CX advises not installing OS updates on Debian 3CX or SBCs. 3CX will install updates they support during an "automatic update" update but not during manual 3CX updates. Not sure about SBC updates since those are not automatic.

pfSense (which is FreeBSD not Debian) forum users have long recommended against Realtek cards/drivers. Myself, I have not had problems, but others do. And of course they work fine in Windows. Not that this really helps you that much, just saying you might find better luck with Intel NICs.

 

[cobaltit]

I think 3CX said OS updates on SBCs were ok at this time because they aren't managing them the same way they do on 3CX PBX installs. I would have just nuked the devices for fresh install or removed the SBC software, done the upgrade, and then reinstalled. That being said, if it does boil down to a driver issue that wouldn't have mattered. We do have a lot of SBCs out there, but not that many in a failover setup but have never had any issues like that with Realtek. As @SteveITS mentions, Realtek is generally frowned upon in firewall applications but that is primarily for high-throughput applications and we've had no issues with Realtek and 3CX or 3CX SBCs.