New to 3CX, concerning first few days

[E N]

We have JUST migrated to 3CX using a hosted service provider we were referred to by 3CX. We did a few weeks of testing with a dozen or so phones without issues, and thought we'd gotten everything worked out, but we have just completed the process of migrating over hundreds of DiDs to this environment, and all hell is breaking loose. In the past few days we've had:

• Unable to call newly ported numbers from the old provider.
• Phones randomly going offline constantly, due to, apparently, overloaded SBCs at the host.
• Random numbers being unable to call other random numbers, both internally. For example I currently can't call one number I'm trying to reach, just get a busy signal, but that person is getting other calls from other people internal and external with no problem. They also say they can SEE my call incoming, but when they pick up I'm not there.
• A global outage "due to a software glitch with our replication software across our nodes. The glitch caused the server's CPU to spike during replication, causing the phones to
  intermittently lose registration"
  We are going to ask the host to install SBCs on prem in our buildings, but I'm wondering if we should be considering alternate hosts preemptively. What is 3CXs process for vetting the hosts they refer users to, and what is the process like migrating from one host to another? We have a seperate trunk provider from the host.

 

[Evolute IT]

Who's the hosting provider they recommended?

Some of those sound like provider issues (SIP trunk) but others sound more like the SBCs are not well speced.

Also, Hosted by 3CX might be better unless you need STUN or SSH access.

 

[E N]

Yes I agree, we are in the middle of trouble shooting the first issue and it sounds like maybe the old provider didn't port the numbers correctly, because calls are coming in fine from other providers/mobile phones.

The SBCs do not sound like they're spec'd appropriately, which is concerning because we've only ported about half our phone so far and only 10% of our staff are actually back in buildings. Do you know why 3CX would have referred us to a third party host when we could have just been hosted with them? When we started this process we submitted the request for details through 3CX's website and our current host is the one who contacted us. For the longest time we actually thought he was WITH 3CX.
I'm hoping on prem SBCs resolve a majority of these issues, but I'm preparing for alternatives and what a host transition might look like.

 

[Evolute IT]

Yes I agree, we are in the middle of trouble shooting the first issue and it sounds like maybe the old provider didn't port the numbers correctly, because calls are coming in fine from other providers/mobile phones.

The SBCs do not sound like they're spec'd appropriately, which is concerning because we've only ported about half our phone so far and only 10% of our staff are actually back in buildings. Do you know why 3CX would have referred us to a third party host when we could have just been hosted with them? When we started this process we submitted the request for details through 3CX's website and our current host is the one who contacted us. For the longest time we actually thought he was WITH 3CX.
I'm hoping on prem SBCs resolve a majority of these issues, but I'm preparing for alternatives and what a host transition might look like.

DM me. I got a few info for you.

 

[SteveITS]

overloaded SBCs at the host

This wording confuses me a bit because the SBC runs in the same network where the phones are, if the 3CX server is not in that same network.

why 3CX would have referred us to a third party host when we could have just been hosted with them?

What size are you (number of SC)? They have limits to the size they can handle.

 

[E N]

This wording confuses me a bit because the SBC runs in the same network where the phones are, if the 3CX server is not in that same network.

What size are you (number of SC)? They have limits to the size they can handle.

We use Cloud Based SBCs located at our 3CX host, they are not local on our network.

We have 800 end users, mostly on desk phones, some with softphones only.

 

[SteveITS]

We use Cloud Based SBCs located at our 3CX host, they are not local on our network.

Is there some sort of VPN so the phones and SBC think they are on the same network? Normally, benefits of an SBC include not sending all traffic over the Internet, for instance extension to extension calls on the same network send audio through the SBC, not out to 3CX.

re: hosting, 3CX supports up to I think 32 SC on their hosting, so you're probably above that.

 

[E N]

No, although they did suggest some kind of VPN tunnel during one of our many phone calls this week troubleshooting the issue. Thats apparently something they're going to try next.
Whatever they're doing on their end doesn't seem to be working, because they've thought things were stable and working every day for 3 days and then as soon as people starting using it again they keep going offline randomly

 

[SteveITS]

Phones connect through an SBC they see on their network and it is just plug and plan. There isn't really a "cloud based SBC" in 3CX so you are in uncharted waters I think...

I would suggest putting the SBC(s) on your network(s) as intended.
https://www.3cx.com/docs/3cx-tunnel-session-border-controller/
https://www.3cx.com/docs/recommended-hardware-specifications-for-3cx/ (SBC section)

Debian/Pi SBCs have a failover option, since you have that many phones: https://www.3cx.com/docs/sbc-high-availability-cluster/

 

[E N]

Interesting, our phone provisioning was done by our 3CX cloud provider (the one 3CX referred us to) and we're just doing what they tell us to do. This isn't some setup we came up with on our own. When I go into the configuration for my desk phone, it doesn't look like the phone is just seeking out whatever SBC is local, it very specifically looks like there is an option for remote SBCs and ours is set up to point to our cloud provider right now.
We also aren't the only ones having this problem, our cloud provider has a message out to all their customers saying that they're having problems with their cloud SBCs and everyone is experiencing the same issues as us, so I get the impression there are lots of people set up and running this way right now.

 

[SweetAction]

It sounds like whatever host you are using is doing their own unsupported 3CX thing with SBCs. 3CX SBCs run in the same network as the phones they are proxying back. Likely your host is doing something with natpass or similar to get sbcs in the cloud. The downside to this is... well you see the downside right now.

For 800 phones you should do a site2site instead of relying on 3CX sbcs which are not really meant for that volume of phones normally. You could do 3CX SBCs by using powerful machines, multiple, and having some phones one some and some on others. Or if desiring a single SBC a very high speced system.

We deal in this space a bit (500+ devices) and there are other ways, some also unsupported by 3CX, to handle this. Can I ask what made you want the phone system offsite vs running it local to the phones?

Also the field they are using for the remote SBC... well its being locked down in u5 as Nick has stated that's not how it's supposed to be used. You won't be able to free-form a SBC address in there and the only options in the dropdown will be the registered 3CX SBCs.

 

[E N]

Thanks for sharing the info, interesting. I assume this "VPN Tunnel" solution our vendor is trying to figure out is the same as the Site2Site thing you're talking about. They are planning a call with me later tonight to explain it, because I don't know what this is going to look like. For the time being we've deployed 3 Core i7 windows laptops to our 3 largest sites and we're going to use those as local SBCs (splitting our phones evenly between them) until we have a permanent solution for this provider, or whatever new provider we decide to go with.

As for why... we wanted to offload dealing with phones entirely to a third party. Over the pandemic we switched to Google Voice for most of our users so people could easily work remotely, but that cost was unsustainable for us, so we started looking for alternatives. It was nice after 20 years to just not have to deal with an internal phone system anymore and we hoped to replicate that experience with a new fully hosted cloud based solution that didn't have us dealing with anything internally on our own. We used this system of connection when testing with around 50 phones for the last few months without issues, so we assumed it could scale, we don't get the impression that the issues we're seeing are limited to us, but they certainly seem to be timed perfectly for when we started to scale our usage.

 

[quadrant]

600 phones is not a large number of endpoints, but it is in that size where the quality of the network deployment will affect performance. You should be fine if you deploy a decent Debian machine with a current i5 processor as a local SBC. If not, it will be quickly obvious and fixable by doubling up the SBCs. The deployment of deep packet inspection, QoS, and ALG will result in no end of unpredictable issues - turn these off. Make sure that the SBC is whitelisted. If your security environment requires DPI, then use a separate VLAN for the SBC and phones and at least turn it off for that traffic.

Direct VPN's and external SBCs begin to get unwieldy above a couple of hundred phones because the real traffic is the constant chatter required to update BLFs. 3CX's proprietary "sbc" manages that update, significantly reducing the messaging across the pipe to the PBX - without it, you'd be pumping a lot of data unnecessarily.

 

[E N]

Update, we've been running on 3 on-prem i5 Windows laptop SBCs for two days and had no issues so far. The few remote phones we still have are still having issues with the cloud SBC, so we're happy we transitioned. I dont know what the vendor's long term solution is going to be, but for now this is an acceptable solution.