Phones lose connectivity after brief outage (SBC).

[centerline]

Hi folks -

This is in the category of "anyone seen this before?"

My little company manages 3CX instances for a good number of nonprofits and churches. Since these organizations get server space donated to them for the most part, we choose to save them some money and host them on Azure or Lightsail. We have SBCs in place for each of them (2 for one instance, one for the other).

At the moment, I have two 3CX instances where I am getting the "up" (and sometimes the "down") e-mail regarding their SBCs frequently - multiple times a day. I think, but I am not sure, that this is an internet provider problem. I'll be checking on that. (If anyone has ideas on that, I'm here for them but it's not the purposes of this post.) It's the only thing these two customers have in common, they are serviced by a local/regional fiber provider that's probably on the same line. One has 3CX hosted on Lightsail, the other on Azure.

Whew, that's the prelude.

The instance with the single SBC (and installed on Azure) loses all of its phones when the brief outage happens - and they never come back. The only way I can get them back up (that I have found) is to restart the SBC. I had the SBC working for years on a virtual machine. Just in case, I moved it to a physical computer a few days ago (a laptop). All of them are using the Debian 3CX SBC iso, as I have found that to be the best option.

I'll get into the logs best I can, but trying to see if anyone has a magic answer or a magic place to start.

Thanks!

 

[centerline]

Ah. Relevant information:

• 3CX Version, e.g. Pro 18.0 Update 9
• Server OS, e.g. Debian 10 / Azure
• Is the 3CX Server Hosted and where? Azure - and yes I know this is self-hosted, but this system is showing as such in the dropdown list for the post.
• IP Phone Make/Model/Firmware version, All phones, doesn't matter what type.
• Provisioning Method: Local / VPN / STUN / SBC ? SBC
• Trunk Provider or VoIP Gateway Make/Model, Comporium (local) Trunk provider
• Has the Firewall Checker passed: YES / NO ? Yes
• Are custom Phone Templates being used: YES / NO ? Only for one phone of the 104.

 

[SteveITS]

"up" notification occurs on every SBC reconnect, no matter how brief. "down" sends if it's been down longer than about 5 minutes.

Is there packet loss when these outages happen?

How many phones are behind the SBC?

 

[SweetAction]

What version is the software running on the SBC?
Does the SBC itself come back up when the internet connection is restored (it will show green or red in the SIP Trunk page)

 

[centerline]

There's no packet loss shown on the SBC stats, but that's RTP. I believe I would need to put something in place to track that otherwise.

104 phones.

SBC Version is 18.1.36, and yes it comes back up as green. Phones just aren't registered after that.

 

[SteveITS]

3CX officially says "up to 100" per SBC: https://www.3cx.com/docs/recommended-hardware-specifications-for-3cx/#h.za60f8obhoix

You might try a continual ping to the 3CX server (ping -n 3600 servername) and see if it records loss in that hour. Also can ping the local router, ISP gateway, etc. and see which has loss or not.

 

[centerline]

I hear you, Steve - and thanks on both. I will just say that this SBC has worked in this configuration for quite some time. There are a lot more phones than traffic - most of these are phones that sit in classrooms in the event of an emergency need. They are rarely if ever used. Probably 25 of them are more regularly used by staff.

I know there's SIP traffic for registration, etc otherwise but the SBC (either as a VM or on separate hardware) isn't breaking a sweat. At all.

I just set up a ping logger very similar to what you suggested. Thanks again!

 

[ivank]

Hello! I suggest collecting SBC logs in verbose mode after it happens again.
The number of phones should not be a problem.

 

[ivank]

Currently I can only imagine this happening if your SBC has two network interfaces, and the one looking outside gets restored fine while the one facing local network stays off.
When you restart the SBC, do you physically reboot the device or just restart the software?

 

[centerline]

When I had it on a VM, which was for about 3 years prior to last week, I would shut down the VM and start it again.

To troubleshoot, I turned that one off and set up a new SBC on a modern laptop computer that just has a bad screen on it. When I had to reboot it today I just did so via ssh (sudo reboot now).

They both have the same issue, I only have one interface active on the VM, and only one on the laptop. They are on a separate voice VLAN for what difference that makes.

I did just turn on verbose mode in the SBC settings. I ought to have plenty of space for that for a while.

I am also having a failure of imagination. I'm not even sure why this is happening, but since I have three locations all on the same ISP all doing it at exactly the same time when it does happen...I'm going with ISP.

 

[ivank]

Any chance SBC loses its IP address temporarily? You can check this if you still got the emails.
Is it statically set or via DHCP? If static, could be another device with the same address?

Bottom line - if the SBC connects well, the only thing that can prevent phones from registering is their inability to reach the SBC. Dig around this.

 

[centerline]

DHCP reservation. No, no change shown in the e-mails. Nice idea, though. I'll dig.

I set up two ping loggers - one outbound from the SBC to the 3CX instance, and one from a linux VM on the same VLAN to the SBC. That won't tell us everything, but might tell us something.

 

[ivank]

Also, try restarting network subsystem of SBC remotely when this happens - let's see if it heals itself.

 

[centerline]

Update, not ready to call it a resolution yet:

Outgoing TCP/UDP packets from our network through our firewall go through a proxy. This has never been a problem before. I configured outgoing 5090 to be explicitly allowed without the proxy. Last time the connection blipped, the phones did not. That's the only time the connection has blipped, so I can't call it a success yet, but one positive thing has happened.

Thanks for all the help, folks.

 

[centerline]

Just to put what I hope is a bow on this...the blips keep happening (but it's not just this system, many of the ones that I support in our area are still doing it) but it's coming back up now. So again, thanks to all who chipped in!