SBC gets disconnected 10-20 times a day

[David_K]

Hello

Since 1 or 2 weeks, we have 2 customers with SBCs getting disconnected 10-20 times a day.

They loose connexion for 1 or 2 minutes, then UP again, then DOWN, UP...

Only 2 customers with the problem and nothing changed on our side.

First customer, SBC is on a Raspberry.
Second customer, a first SBC on a dedicated Windows machine + a second backup SBC on a Windows Server. And when 1st SBC gets disconnected, 2nd one also.

Already restarted the 3CX VMs that are hosted at VULTR, and their support told nothing wrong with the hosts or switches.

 

[Evolute IT]

Hello

Since 1 or 2 weeks, we have 2 customers with SBCs getting disconnected 10-20 times a day.

They loose connexion for 1 or 2 minutes, then UP again, then DOWN, UP...

Only 2 customers with the problem and nothing changed on our side.

First customer, SBC is on a Raspberry.
Second customer, a first SBC on a dedicated Windows machine + a second backup SBC on a Windows Server. And when 1st SBC gets disconnected, 2nd one also.

Already restarted the 3CX VMs that are hosted at VULTR, and their support told nothing wrong with the hosts or switches.

What DC in Vultr? We had some networking issues with them for a while and we moved over to DO because of that. No issues ever since.

 

[David_K]

OK I think I found a clue : the only 2 customers we have problems with, have FQDN like xxxx.3cx.fr (when we installed them, this was the form of the FQDN we got)
So since only those 2 have a problem, I suspect DNS issues on the 3cx.fr resolver

 

[Colorado VoIP]

when we see this with customers it's usually their internet flapping down/up.

 

[David_K]

no issues with Internet at both customers

 

[David_K]

What DC in Vultr? We had some networking issues with them for a while and we moved over to DO because of that. No issues ever since.

We use DC Paris
We almost never have issues and all other VMs are also in DC Paris, without any issues

 

[SteveITS]

suspect DNS issues on the 3cx.fr resolver

Are they Enterprise? Those have a 5 minute TTL but Pro is 6 hours.

ISP issues that cause momentary drops are not usually detectable for TCP connections, which retry lost packets.

 

[David_K]

Are they Enterprise? Those have a 5 minute TTL but Pro is 6 hours.

ISP issues that cause momentary drops are not usually detectable for TCP connections, which retry lost packets.

They are PRO, but what will it have to do with TTL?
I don't get it.

 

[David_K]

A user just told me he lost connexion with his Web app softphone, working from home (so no SBC involved)
This confirms a bit more my doubts that there is an issue with the 3cx.fr DNS resolver

 

[SteveITS]

what will it have to do with TTL?

I would think if it was a DNS issue the SBC would connect and be fine for 6 hours, then try to look up DNS and fail, and disconnect at that point.

In my travels I have seen odd issues such as both ends of the connection being perfectly fine, but some backbone ISP in between has a routing problem that causes dropped packets. A traceroute and/or continually pinging all the hops can help pinpoint that.

 

[David_K]

I would think if it was a DNS issue the SBC would connect and be fine for 6 hours, then try to look up DNS and fail, and disconnect at that point.

In my travels I have seen odd issues such as both ends of the connection being perfectly fine, but some backbone ISP in between has a routing problem that causes dropped packets. A traceroute and/or continually pinging all the hops can help pinpoint that.

Yes but problem is that the disconnexions lasts 1 or 2 minutes each time maximum so the time we do a tracert problem is solved

And what's strange is that customer A and customer B both gets disconnexions at the same time.
They have different servers, different telcos... the only common point I can see at this moment is that they use both an FQDN on domain 3cx.fr (they are the 2 only)

 

[ivank]

I would think if it was a DNS issue the SBC would connect and be fine for 6 hours, then try to look up DNS and fail, and disconnect at that point.

SBC resolves FQDN only upon new connection and thus will not break existing connection unless it is broken. Which almost always happen due to dropped packets on the route.

 

[David_K]

SBC resolves FQDN only upon new connection and thus will not break existing connection unless it is broken. Which almost always happen due to dropped packets on the route.

OK so as soon as connected it won't regularly refresh the connection by requesting again the DNS?
I thought so.

At customer 1 we did reboot the telco router, and seems to get no alerts anymore. So could be a telco issue.
We did the same at the customer 2 and still get the alerts...

 

[Lee Cramman]

Remember, commonality doesn't always equal causality so keep an open mind. Also, different telcos doesn't always = different telcos - a lot of small telcos are simply resellers, rebranding another firms product as their own.

It's almost always broadband connectivity when SBCs do this. It's difficult to diagnose because often you just see a bit of a pause on the network (so unless you're looking at exactly the right time you don't notice it). Sometimes the router doesn't even report a disconnect (although often it will show a resync event in the logs).

My first port of call would be the router logs for the time of the outages to see if anything looks out ofplace. If that doesn't bear fruit setup a ping every few seconds to your PBX piped to a log file and see if pings drop at the same time as your SBC goes down. Telcos often have access to very good diagnostic information - if you raise a support case showing the times of your issues they will be able to tell if their were any unusual reconnection / resync events from their end.

You could also try spinning up an SBC at another test location and see if it loses connectivity at the same time. If the sites SBC goes down and your test location doesn't that gives you some pretty big clues.

 

[David_K]

You could also try spinning up an SBC at another test location and see if it loses connectivity at the same time. If the sites SBC goes down and your test location doesn't that gives you some pretty big clues.

That's a good idea

I did activate a monitoring from outside to the hosted PBX + a monitoring from inside (from the SBC itself) both every minutes.
And when I get SBC disconnections alert, I don't get the monitoring alerts so it's like everything was OK...

 

[Lee Cramman]

Was the monitoring you did from the SBC to your VULTR hosted PBX (or the same in reverse) or simply to check internet connectivity? If it was only internet connectivity I would repeat between SBC / PBX. It's quite possible for VULTR to have issues.

I know this is a silly question but... PBX is on a fixed IP with VULTR, yes?