Will Debian 13 fix the SBC HA Cluster Breakage Issue?

[Gavino]

In reference to the SBC High Availability docs and quoting from the last bullet point of "Known Issues and Limitations"

• If local network connectivity is temporarily disabled or broken for any of the member nodes, the clustering service may not recover when connectivity is restored. Ensure network connectivity between the member nodes, to avoid causing nodes to assume the active role concurrently and activating the cluster resources on both nodes. This issue is to be resolved in a subsequent Debian / Raspbian Linux version.

I have this exact issue when doing Ethernet switch firmware updates. A couple of weeks ago I did a switch firmware update, and both SBCs in a HA pair were not able to be logged into after the switch successfully updated and came back - either via the SBC unique IPs, or the floating shared IP, even when shutting switch ports to try and isolate one to log into. In the end I had to get remote hands to power-cycle both SBCs, one after the other.

Then on the weekend just gone there was another switch firmware update. I thought I'd be clever this time and schedule a "shutdown -r <time>" putting in a time about 5 minutes after the switch would be back online. I'm not sure why, but only one of the SBCs recovered. Thankfully, since I had one working SBC in the pair, I shut the network port on the failed one, and will go to site myself sometime this week and unshut the network port and reboot the failed/locked SBC. There's no serial console port on these, so I'll just power-cycle and it should come back after that.

These SBCs are on Raspberry Pi 3B+ (the official ones from Pimoroni supplied a few years ago) and when I upgraded to 3CX v20, I did a fresh wipe of these SBCs and built them from scratch following the guides to the letter, with no other extraneous software installed. I do regular apt update/upgrade/autoremove on them to kept the packages fresh. "cat /etc/debian_version" currently shows Debian version 12.11.

In the quote above, the key thing for me is "This issue is to be resolved in a subsequent Debian / Raspbian Linux version." Well Debian 13 was released on the weekend just gone, and I have been busy updating some other Debian 12 installs to version 13 and that has gone smoothly (these servers have nothing to do with 3CX). Can anyone answer the following?

1. Will my 3CX SBC HA pair (arm64 / Raspberry Pi 3B+) still work if I do a full upgrade to Debian 13? And,
2. Will upgrading to Debian 13.0 fix this issue?

If there's a sure-fire way to recover from this in Debian 12, or at least avoid it when doing switch firmware updates/upgrades - then more than happy to stick with Debian 12, until it is no-longer supported. At the end of the day, all I want is a workable fix, so I don't have to use remote hands or go to site myself, every time I do a switch firmware update! Thanks in advance, Gavin O.

 

[Gavino]

For anyone interested and lurking, I did the following on my backup SBC in a HA cluster..

crm node standby
systemctl stop 3cxsbc
systemctl stop corosync
systemctl stop pacemaker

I then did the switch firmware update (an update of a Ubiquiti EdgeMax EdgeRouter switch from 1.11.0-lite to 1.11.1-lite using a UISP-Console hardware device, and the UISP application).

I thought that might solve the issue since the backup SBC was on standby and services shut down. When the switch came back after successful firmware update, the primary SBC was inaccessible. Ether via permanent IP or static IP - I couldn't get into it remotely at all. Frozen. Damn. Fortunately I could bring the backup SBC online ("crm node online", then systemctl to start the services). I rebooted the backup SBC as well to make sure all was still fine after a reboot, which it was.

Possibly I have to stop the listed services on both SBCs before doing any future switch firmware updates? I thought that just offlining the services on the backup SBC would be enough, but turns out it wasn't. Hopefully doing a power cycle of the inaccessible primary SBC is enough to bring it back online.