Recent issues with 3CX Hosted

[amayo]

I’m writing because one of my clients has been experiencing phone system downtime recently, and I’ve been unable to get solid answers as to what is happening during the outages through 3CX support.

The issue is as follows:

The customer has 2 sites in 2 different cities in Ohio. One in Columbus and one in Cincinnati. Twice this month – both sites have gone offline and users connected to the company network in both locations cannot reach “XXXX.oh.3cx.us” which is their phone system URL. The SBCs at each site go offline within seconds of each other, the web client becomes unreachable from a browser and softphone apps cannot connect either. This leaves the phone system completely unusable from both sites. However, while the employees are experiencing the issue, all other Internet is perfectly accessible. They simply cannot reach “XXXX.oh.3cx.us”, yet I can reach the phone system URL from outside their network and from basically any other network except theirs. Users who are at home can use the soft phones apps without issue.

Additionally – while experiencing the issue, it is obvious to me that there are no blacklisted IP addresses listed on the phone system – so that is not the issue. The issue does seem to resolve itself after some time. Once it took over an hour, and the more recent time it corrected itself within about 25 minutes. However, this is unacceptable.

3CX support has not been helpful, claiming that since it is not a hosting issue and a networking issue instead – that they cannot help. However, I can’t fend for myself if the networking issue resides on the hosting provider’s side, or at least I don’t know how to do that.

Both sites use the same ISP – (Spectrum Business) – but are located in different cities and have no connection with one another. The only real common denominator is they both need access to the phone system URL and get dropped from it simultaneously. Any insight would be very much appreciated as I’ve got to find a way to stabilize this, even if it means moving away from 3CX altogether.

 

[ThomasD_3CX]

@amayo

Please allow me to be more precise in regards to the incidents in reference.

You contacted us reporting the above issue twice, once on 27th Aug 2022 and once on 9th Nov 2022.
In both cases you have reported that the 3CX Hosted system was not reachable although we were able to access the servers with no issue. Both times we contacted you via phone to further investigate the issue. We were running captures on our side but unfortunately the requested info was not provided to us, for the connectivity issue on the remote site (your site)

Furthermore, your statement:

"They simply cannot reach “XXXX.oh.3cx.us”, yet I can reach the phone system URL from outside their network and from basically any other network except theirs. Users who are at home can use the soft phones apps without issue."

verifies the fact that our servers were not down/unreachable and this was also communicated to you via email but also over the phone during our communication.

If our service was impacted, you would not be able to connect from any location, which was not the case as you have stated above, but you had also verified during our live conversation, when you were able to reach/contact the 3CX Hosted System the same time the remote location in question could not.

As you understand this also rules out any host connectivity issue as well.

I would recommend contacting the local network administrator, to monitor the traffic on the remote site network level, from/to the 3CX hosted system so you can identify the connectivity issues you are facing.

 

[cobaltit]

The part that is missing is you. Partners are supposed to fill in the gaps between 3CX and their environment.

For troubleshooting, if you are on 3CX hosted, you can check the instance status in the partner porta (not sure if you have access at the trainee level) or via the customer portal:



Next step is 3CX typically uses Digital Ocean so it's a good idea to bookmark the status pages. For Ohio, you are probably on the NYC data centers. From the DO status page, it looks like there were events on 11/13 and 11/17 that may be related to what you mentioned:



It's possible it's just a networking issue between Spectrum and the DO instance. You don't mention if the test points outside their office were also on Spectrum but if it worked from non-Spectrum connections but not from Spectrum connections then that is not a 3CX issue. Another possibility is if both clients share the same firewall manufacturer or security stack, perhaps something that handles DNS like Cisco Umbrella and that's why they both get blocked. You can also do a trace from each location to the 3CX instance to see what the path looks like (final hops won't work as 3CX hosted instances do not respond to ICMP) but then you can compare when it doesn't work to see if it's getting stopped in before it hits DO or not.

 

[VoIPTools]

I did not read through all the above. But I will make a quick observation. This sounds to me like a internet service provider DNS/routing issue. I have experienced this very same issue with a internet provider. It turned out that the internet provider was having routing issues in their internal infrastructure. I concur with 3CX that if you can get to 3CX from other locations, then it probably is not their fault. I would start asking questions of your internet provider.