Multiple SBC's on Raspberry Pi no longer connecting

[RobPounds]

Hi people. Hoping someone can help point me in the right direction as I feel I've tried everything.

We use an on-premise 3cx server v16 with latest updates applied.
I have two Raspberry Pi SBCs at two different remote locations (work from home). Both have been offline (last successful registration was identical to within a few seconds) for two weeks now.
Each site has been working without issue for a long time.

We had a local firewall issue and needed to re-boot our firewall. This has since been reset and the stored config file re-applied.
Have subsequently rebooted our entire network and servers.

We are seeing absolutely no hits on the firewall from the remote connections for ports 5060 or 5090
I have run the 3cx firewall checker - OK.
I have remote connected to each remote location PC and manually sent packets on port 5060 and 5090 from those locations - OK (packets received in firewall log and forwarded to 3cx server). It was necessary of course to reconfigure the web browser to allow traffic out on port 5060 and 5061, before browsing to our 3cx FQDN on those ports.
I have checked the windows firewall on the 3cx windows server and the inbound ports appear to be correctly configured (certainly the above ports are allowed through). Again, I'm not seeing the traffic at the firewall log, so suspect the 3cx server setting is not relevant.

I have not yet logged onto each Pi as remote access was disabled. Again, both have simultaneously gone offline at the same time.

I have a niggling concern that maybe the Pi's have both been upgraded to v18. Is there a way to verify if this update was deployed without accessing the Pi itself (eg a log file somewhere with any update detail)? Assuming a v18 sbc won't work with a v16 system?

If this turns out to be the case, what is the best way to restore v16 on the Pi?

But again, absolutely no inbound hits on the firewall from the SBC. This, I don't understand as surely even if the SBC had been upgraded to v18, I should see packets arriving at the firewall?

TIA



Rob

 

[NickD_3CX]

Hi Rob,

From your description, the 2 RPi's were installed with the SBC software. This software does not auto-update, unless you prompt it to via the 3CX Management Console, or if you have done any cronjobs on the Pi's themselves.

This aside though, even if this were the case, I think you wold still some some traffic hitting the Firewall, which in your case, you say you don't.

If the Firewall Checked comes back green, then you can be fairly certain that for UDP everything is OK, but can you also double-check that you have opened the Tunnel Port (default 5090) for TCP as well?
Remember, Tunnel uses both.

If you have checked all of the above, then I'm afraid the next thing would have to be an on-site visit, if you don't have remote access to the Pi's to run some tests while SSH'd onto them.

 

[Alphabetic]

Since the problem occurred after you had a "local firewall issue" then this is where your suspicion should stem from.

Do you have a static IP on your WAN? Did rebooting give you a new WAN IP?

What DNS did you set up on the PI and did you use FQDN when setting up the PIs?

Have you asked them to reboot the Pis?

If you install Windows SBC on their PC does it connect?

Get on their PC again, SSH in to the Pi and just remove reinstall 3CX and set it back up.

 

[RobPounds]

@NickD_3CX & @kieferschild

Thank you both for the considered effort you have both put into your replies. I have another Pi which i will be configuring and testing on our DMZ before shipping to one of the sites. Replies to your thoughtful questions as below. Thanks again to you both Will update once i have reviewed again.

"This software does not auto-update, unless you prompt it to via the 3CX Management Console"
- this is the niggling doubt that i might have been a complete prat here I might have inadvertently done this simultaneously to the fallout from the initial firewall issue without recognising the difference in versions. However does not explain the lack of packets at the firewall.

"can you also double-check that you have opened the Tunnel Port (default 5090) for TCP as well?"
- Yes, this is done and double checked since the firewall reboot.
- Was a working system before.

"Do you have a static IP on your WAN? Did rebooting give you a new WAN IP?"
- Yes to static IP. No to new WAN ip.

"What DNS did you set up on the PI"
- Good question. Will check but suspect the main gateway of the remote worker.

"did you use FQDN when setting up the PIs?"
- Yes.

"Have you asked them to reboot the Pis?"
- Oh yes

"If you install Windows SBC on their PC does it connect?"
- Good call. Did not try.

 

[RobPounds]

If the Firewall Checked comes back green, then you can be fairly certain that for UDP everything is OK, but can you also double-check that you have opened the Tunnel Port (default 5090) for TCP as well?
Remember, Tunnel uses both.

@NickD_3CX, can you kindly advise detail on packet flow here?

I've re-checked and can confirm hits at the firewall this morning on port 5090 from one of the IP addresses that has an SBC. The packet is forwarded correctly to the 3cx server. Would be a fair assumption that until the tunnel is established, the SBC won't send traffic on 5060? Assuming I have stupidly hit "upgrade SBC" to v18, and the Pi now runs V18, I am thinking that V18 won't establish a tunnel with v16 server, and so therefore we won't see the Pi register with the server and therefore any traffic on 5060?

 

[Alphabetic]

@NickD_3CX, can you kindly advise detail on packet flow here?

I've re-checked and can confirm hits at the firewall this morning on port 5090 from one of the IP addresses that has an SBC. The packet is forwarded correctly to the 3cx server. Would be a fair assumption that until the tunnel is established, the SBC won't send traffic on 5060? Assuming I have stupidly hit "upgrade SBC" to v18, and the Pi now runs V18, I am thinking that V18 won't establish a tunnel with v16 server, and so therefore we won't see the Pi register with the server and therefore any traffic on 5060?

the 3CX traffic (SIP & RTP) goes via the tunnel 5090

https://www.3cx.com/docs/3cx-tunnel-session-border-controller/

I'm not even sure its possible to upgrade the SBC to v18 if your PBX is not 18 - At least I cannot do that from the v16 console Im looking at.

It's not something stupid like the IP has been blacklisted in 3CX?

 

[RobPounds]

the 3CX traffic (SIP & RTP) goes via the tunnel 5090

https://www.3cx.com/docs/3cx-tunnel-session-border-controller/

I'm not even sure its possible to upgrade the SBC to v18 if your PBX is not 18 - At least I cannot do that from the v16 console Im looking at.

It's not something stupid like the IP has been blacklisted in 3CX?

@kieferschild
Thanks for your extremely fast response . Yeah checked that too .

I just have the vaguest of memories (it's not that good on a good day to be honest) that there was an option to upgrade the SBC to v18 and without thinking, I possibly did that.

But interesting it is not an option on the system you're looking at.

I'll carry on reinstalling on a fresh Pi, test again and report back.

 

[RobPounds]

If i can get logged into the SBC, is there a way to identify the installed version of 3cx?

 

[NickD_3CX]

If i can get logged into the SBC, is there a way to identify the installed version of 3cx?

Sure, you can use this command:

dpkg -l | grep 3cxsbc

 

[RobPounds]

Sure, you can use this command:

Thanks again @NickD_3CX. Whilst looking at some other options, I found a spare SBC on a pi, but using your link above, it has v15. I'm struggling to find on the website how to manually update it (it won't connect to the 3cx server). Would you be so kind as to let me know how please?

 

[NickD_3CX]

To be honest, I think your best bet would be to uninstall the SBC you current have given that it's this old (you hsould have upgraded it earlier), then just install the newest version.
You can do this with these commands:

sudo apt remove 3cxsbc --purge
wget https://downloads-global.3cx.com/downloads/misc/d10pi.zip; sudo bash d10pi.zip

 

[RobPounds]

To be honest, I think your best bet would be to uninstall the SBC you current have given that it's this old (you hsould have upgraded it earlier), then just install the newest version.
You can do this with these commands:

sudo apt remove 3cxsbc --purge
wget https://downloads-global.3cx.com/downloads/misc/d10pi.zip; sudo bash d10pi.zip

You're a diamond Thanks so much. Will now look at the other units with the above info in mind.

 

[RobPounds]

@NickD_3CX
Actually we just found the issue with the original Pi after sticking a screen on it.
It won't boot. something went very wrong, but on two Pi units simultaneously.
Result of an upgrade from 3CX as I thought?
We didn't manage to check the other Pi that went offline at the same time yet, but I imagine it is a strong possibility that it has the same problem.

 

[NickD_3CX]

Glad the uninstall/re-install instructions worked!

Onto the boot failure, doubtful. The SBC upgrade when triggered from the Management Console just upgrades the specific '3cxsbc' package, not anything else of the OS.

The only way this could have happened is if you put any other software on there that automatically installs Debian updates.

If this is not the case, then despite the coincidence that 2 went offline around the same time, I can't think of something else other than this being a coincidence (assuming that the second PI is doing the same, this will are assuming...).

 

[RobPounds]

Glad the uninstall/re-install instructions worked!

Onto the boot failure, doubtful. The SBC upgrade when triggered from the Management Console just upgrades the specific '3cxsbc' package, not anything else of the OS.

The only way this could have happened is if you put any other software on there that automatically installs Debian updates.

If this is not the case, then despite the coincidence that 2 went offline around the same time, I can't think of something else other than this being a coincidence (assuming that the second PI is doing the same, this will are assuming...).

I will get eyes on the other SBC to confirm, probably tomorrow.
Will update when i know more.

Thanks again for your help.

 

[RobPounds]

Just a final update.
On our second faulty SBC, I found that unlike the first which had a faulty Pi OS, this one booted without problem.
I checked the version and it is still v16 (which matches our server).
The problem was found to be missing value settings from the 3cxsbc.conf file, namely for the Tunnel address.

Specifically:
TunnelAddr=
PbsSipIP=
Above entries were as typed above, but both had missing values.
Provisioning Link entry read:
ProvLink=sbc/[our correct auth link]
again, here essentially the tunnel address and port value were missing..

After manually re-instating the correct data at each of the above 3 lines, the SBC immediately connected with our server.

I have no explanation for how this data was removed from the file and no-one else has access to the Pi

Finally, to address the doubt I had about a v18 SBC connecting to a v16 server, after removing 3cxsbc from our spare pi and downloading the 3cxsbc from the link that @NickD_3CX provided above, I established that v18 was installed which for the benefit of anyone who reads this, does indeed work without issue with a v16 server.

@NickD_3CX, any ideas why the 3cxsbc.conf file would have those entries removed?

Finally, my thanks once again to both @NickD_3CX and @kieferschild for their fast replies, and kind information which ultimately led to a restored system in a short time.

 

[NickD_3CX]

@NickD_3CX, any ideas why the 3cxsbc.conf file would have those entries removed?

I'm glad everything is sorted out!

Unfortunately I can't think of a reason how the contents of this file would change, especially if no one had access to the PI.