SBC more than hundred lines trunk UP

[AWS2P]

Hi
today I received near 200 hundreds email notifications for 2 SBCs on 2 different PBXs. PBX are latest release and SBC are Windows 10 versions. PBX are cloud hosted on AWS.

ISP line got no change during the night and line is Fiber with huge bandwith.



Messages started at 4h32 AM and stopped 5H32 AM (for this one I got 120 identical notifications)


For second PBX there are 3SBCs linked for 3 locations (so 3 different ISP) 2 are Debian , 1 Windows
All the messages are same as above trunk SBC has changed to UP.

Messages started at 2h47 AM and stopped 5H47 AM (for this one I got 178 identical notifications)

Only common point is AWS for hosted PBXs

Few days before I got also a lot of DNS resolution/ Network failure messages (related to AWS/OpenIP trunk) a lot of French users related same thing to OPENIP and 3CX support.
Registration at OpenIP has failed.
Destination (sip:94.143.87.216:5060;lr) is not reachable, DNS error resolving FQDN, or service is not available.

Any ideas what can I do ?

 

[StefanW]

I can only guess as there is nothing 3CX related, we also monitor in the last days that we have a lot of lost packages to destinations in France. Maybe some BGP routes are affected which causes this.

 

[AWS2P]

What means BGP ?

 

[AWS2P]

on french forum Several 3CX partners are complaining about Openip / AWS / 3CX trunk randomly disconnected and audio one way when in call or turning really bad, but this is not permanent and customers are with good ISP line , most fiber.

 

[AWS2P]

For SBCs I don't understand your answer, this is a 3cx device connected to 3cx PBX with an internet line in between, so why do i receive every minute a notification to say SBC is UP, never got one to say it was down.

I need a little more explainations.

 

[NickD_3CX]

Hi @aws2p
Let me elaborate a bit, what we suspect is that since yesterday, we have seen similar cases reported to us across some Hosting Providers (like AWS) that have unexplained packet loss.
This is no something to do with the 3CX software itself and we suspect this has something to do with the internet traffic to/from the hosting Provider(s).

That is where the "BGP" comment comes in, meaning that the routing from the Hosting Provider(s) is what is causing this.

FYI, BGP: https://en.wikipedia.org/wiki/Border_Gateway_Protocol

 

[AWS2P]

Thanks Nick for detailled answer.

 

[AWS2P]

To complete the board, what is the trigger for SBC or PBX to send an email notification for SBC trunk UP or DOWN as I said I receive every minute and sometimes up to 3 notifications in same minute.

 

[Evolute IT]

To complete the board, what is the trigger for SBC or PBX to send an email notification for SBC trunk UP or DOWN as I said I receive every minute and sometimes up to 3 notifications in same minute.

Run a packet capture when it happens and send it to me. I am curious, but to me it sounds like a bad DNS or, as said above, BGP issues at the ISP level.

 

[NickD_3CX]

To complete the board, what is the trigger for SBC or PBX to send an email notification for SBC trunk UP or DOWN as I said I receive every minute and sometimes up to 3 notifications in same minute.

What Frederick said, all email notifications are sent from the PBX, and the trigger is what you would expect, when the SBC disconnects and reconnects.

This is also another symptom of what we have been hearing the last 1-2 days, and what we have noticed so far is that when there is an issue with the SIP Trunk (e.g. goes unregistered), the SBC will very likely also go unregistered.

This is what leads us to believe, as Frederick pointed out, that it must be some routing issue on the ISP level.

 

[AWS2P]

Today a French user of 3CX got an interesting answer from OpenIP SIP provider, see below :

Good morning all
I enclose below the response of OpenIP technical support relating to the problem of deregistering the trunk sip:

"Following the various feedbacks that we had and thanks to the sending of traceroute, we were able to identify a common point between all the impacted PBXs: they are all hosted in German data centers.
We opened a signal to our internet forwarder who confirmed to us that the interconnection between France and Germany is subject to disruption.
They tell us that they have cut this interconnection so that the traffic passes through another path.
We have restarted traceroutes and no longer see any increase in latencies. "

Have you seen an improvement on your side for your virtual machines hosted by 3CX in Digital Ocean data centers in Germany?

On my side all problems seem to have been stopped. (using AWS and not DO)

 

[NickD_3CX]

We are also continuing to monitor this and indeed our investigation is also showing that it has improved a lot since Friday, but there is still some small amounts of packet loss (~20% loss on Friday VS ~2% today).

I am happy to hear that OpenIP found the issue and their up-link provider is working to resolve it.

 

[AWS2P]

I can see today some sbcs reconnection , perhaps 2 or 3 times a day, is the SBC check very sensitive to send notifications ?
What I don't understand is, trunk is not disconected , while SBCs are, is it specific process for SBCs?

 

[NickD_3CX]

Yes, SIP Trunks register over UDP every 2-3 minutes, if there is a "glitch" in between the registration attempts, the trunk will remain registered, you may only notice it on an active call where the audio may become a bit funny for 1-2 seconds.

On the other hand, the SBC keeps a constant connection to the PBX over TCP, and while TCP does have mechanisms to deal with some packet loss, if a connection reset is caused, you will still get the email.
The good thing though is that due to TCP, the SBC can immedaitely detect the dropped connection and immediately reconnect, causing minimal downtime.

So to answer your question in short, no, SBC is not more sensitive, it is just different ways of connecting.

 

[AWS2P]

Thanks Nick you are a good teacher , your explainations are crystal clear