Voip.msTrunks Keep Disconnecting/Reconnecting

[SethTalley]

As referenced here: https://www.3cx.com/community/threads/3cx-v18-trunk-keeps-disconnecting-telnyx.127155/post-605319

We're running 1ea VoIP.ms, Telnyx and Flowroute. The outages usually last between 2 and 10 minutes, at least according to the software - to the best of my knowledge they haven't interfered with our calls. BUT since we're a midwifery practice and we need to know whether our patients can page us in the middle of the night, five midwives (including my wife) and three students received push notifications at 11:56, 11:58, 12:01, 12:03, 12:06, 12:08, 12:21, 12:23, 1:03 and 1:05am last night.

This obviously isn't making me popular.

Last night it was all Voip.ms, but it'll often be two in a day, and once was all three. I would say it happens once or twice a week.

What can I do to fix this, please?

 

[Guillaume Bourgeois]

Voip.MS says they have no minimum but they recommend setting the parameter to 300 seconds. Telnyx says their supported minimum is 60 seconds.

So that should be fine—3CX v20 is configured to re-register every 60 seconds.
This means 3CX re-registers before the 60-second expiration, typically around 10% earlier. In other words, approximately every 54 seconds, 3CX sends a new request to the provider to "extend" the session for another 60 seconds.
Read.: Learn more about how 3CX refreshes the registration

In any case, if the provider doesn't support this interval, they will return a different value in the Register confirmation (200 OK) and 3CX will automatically adjust to the value returned by the provider. Now that we know the timing is supported by providers, we can compare it with the 3CX logs you shared.

This mechanism does not appear to be faulty, as we can see in the logs you shared with us the frequency of re-register, greater than 60 seconds, and the vendors have confirmed that "60 seconds" is accepted.

• The next step will likely be the most complex.
  Access the Activity Logs in 3CX, go to Settings, change the logging level to Verbose, and save the changes.
  
  Keep this setting as is for a few days.
  
  
  
  
  
  
  
  The next time the issue occurs with any of your providers, as soon as possible after the issue resolves (on the same day), go back to your 3CX PBX and download the Support Info.
  It's crucial to do this on the same day as a registration failure, as the logs are purged daily.

Please note the time and approximate duration of the SIP Trunk outage to allow me to filter the data as much as possible.

Once you have a SIP Info package that contains the history (from the same day) of a SIP registration failure,
I would appreciate it if you could send me in a private message the 3CXPhoneSystem-*.blrec and 3CXPhoneSystem.bldef files from the compressed Support Info folder. I don't need the entire Support Info folder.

With this files, be aware that I will have access to a significant amount of informations.
You can learn more about what’s included in these files by visiting this page : https://www.3cx.com/docs/3cx-log-viewer/
Proportionately, I will have access to filter the information for you to clarify the situation, and you will have everything you need to seek assistance from the appropriate party.

The history / this files does not contain any passwords or audio recordings.
Only ; Phone numbers, usernames (without passwords), public IP addresses, etc.
However, out of respect for you, even though the information in these files may seem trivial to many, I believe it’s important that you are fully informed and understand what I am asking you to share with me privately, so that I can assist you effectively.

I hope I can help you !

 

[SethTalley]

Curses. I missed the "set it to verbose" part until now - I had one at 8 this morning. Will be in touch as soon as I have another. Thanks, Guillaume.

 

[pellisCMIT]

Been following this thread, having the same issue, it went down last Monday twice relatively close together, nothing since, but I do have verbose captures for the incident, I couldn't make anything from it though, just seems like it loses connection.

I'm now running an nmap probe each minute to check the SIP stats of the VOIP.ms server I'm connected to, to see if I lose connection at the same time the 3CX Cloud loses it's connection. That will clue me in to it being a VOIP.ms problem rather than a 3CX Cloud problem.

 

[Evolute IT]

Been following this thread, having the same issue, it went down last Monday twice relatively close together, nothing since, but I do have verbose captures for the incident, I couldn't make anything from it though, just seems like it loses connection.

I'm now running an nmap probe each minute to check the SIP stats of the VOIP.ms server I'm connected to, to see if I lose connection at the same time the 3CX Cloud loses it's connection. That will clue me in to it being a VOIP.ms problem rather than a 3CX Cloud problem.

VoIP.ms has issues recently. We've seen many trunks go up/down intermittently on various systems, all hosted in different cloud providers. Pretty sure it's a VoIP.ms problem.

 

[pellisCMIT]

I'm moving to another provider as a result of this but I wanted to point out that in my so far one sample set, voip.ms goes down only for 3CX, not other locations.
I'm connected via Oracle DC in Atlanta and a scripted continuous nmap probe of voip.ms Atlanta server showed SIP was responsive during the period that 3CX was unregistered:
# Nmap 7.95 scan initiated Thu Sep 26 02:27:32 2024 as: "C:\\Program Files (x86)\\Nmap\\nmap.exe" -e eth1 --script=sip-methods -sU -Pn -oG grep.txt --append-output -p 5060 208.100.60.17
Host: 208.100.60.17 () Status: Up
Host: 208.100.60.17 () Ports: 5060/open|filtered/udp//sip///
# Nmap done at Thu Sep 26 02:28:04 2024 -- 1 IP address (1 host up) scanned in 31.83 seconds
# Nmap 7.95 scan initiated Thu Sep 26 02:30:04 2024 as: "C:\\Program Files (x86)\\Nmap\\nmap.exe" -e eth1 --script=sip-methods -sU -Pn -oG grep.txt --append-output -p 5060 208.100.60.17
Host: 208.100.60.17 () Status: Up
Host: 208.100.60.17 () Ports: 5060/open|filtered/udp//sip///
# Nmap done at Thu Sep 26 02:30:41 2024 -- 1 IP address (1 host up) scanned in 36.33 seconds


Last night at 2:26 AM EST:


I'll submit this to voip.ms as well but someone's network security might be momentarily blocking one side or the other, since only multiple 3CX hosted instances can't seem to reach the SIP, I'm still thinking it's something on the 3CX side but I can't really continue to fight with this so we're going elsewhere. I am curious if the outage times I'm seeing match anyone else with 3CX+Voip.ms.

 

[Evolute IT]

I'm moving to another provider as a result of this but I wanted to point out that in my so far one sample set, voip.ms goes down only for 3CX, not other locations.
I'm connected via Oracle DC in Atlanta and a scripted continuous nmap probe of voip.ms Atlanta server showed SIP was responsive during the period that 3CX was unregistered:
# Nmap 7.95 scan initiated Thu Sep 26 02:27:32 2024 as: "C:\\Program Files (x86)\\Nmap\\nmap.exe" -e eth1 --script=sip-methods -sU -Pn -oG grep.txt --append-output -p 5060 208.100.60.17
Host: 208.100.60.17 () Status: Up
Host: 208.100.60.17 () Ports: 5060/open|filtered/udp//sip///
# Nmap done at Thu Sep 26 02:28:04 2024 -- 1 IP address (1 host up) scanned in 31.83 seconds
# Nmap 7.95 scan initiated Thu Sep 26 02:30:04 2024 as: "C:\\Program Files (x86)\\Nmap\\nmap.exe" -e eth1 --script=sip-methods -sU -Pn -oG grep.txt --append-output -p 5060 208.100.60.17
Host: 208.100.60.17 () Status: Up
Host: 208.100.60.17 () Ports: 5060/open|filtered/udp//sip///
# Nmap done at Thu Sep 26 02:30:41 2024 -- 1 IP address (1 host up) scanned in 36.33 seconds


Last night at 2:26 AM EST:
View attachment 44139

I'll submit this to voip.ms as well but someone's network security might be momentarily blocking one side or the other, since only multiple 3CX hosted instances can't seem to reach the SIP, I'm still thinking it's something on the 3CX side but I can't really continue to fight with this so we're going elsewhere. I am curious if the outage times I'm seeing match anyone else with 3CX+Voip.ms.

Yup couple of our customers. Same thing around the same time.

 

[pellisCMIT]

Were they all 3CX hosted or a mix of onprem and hosted?
I'm still just testing out our NFR instance so I'll message my rep, but I can't submit a ticket on this (which seems crazy when you're 'kicking the tires' and running into problems)
If it's all hosted, and if you care to, maybe you could raise a ticket and direct them to this thread, I would hope as a supported SIP trunk, there is some communication channel between the two that they can try to figure out which side or route is causing the problem.

If it's happening on a mix, then I'll assume it's just voip having a bad route on some paths and not others.

Thanks for everyone's help in this thread, I really appreciate it!

 

[rayfield]

I'm seeing something similar to this, with one customer, who is hosted by 3CX. The connections that keep dropping, though, are SBC connections.

They have two 3CX SBCs running, one at the main office in SW Missouri and one at a satellite office in St. Louis. Both SBC connections will drop at various, random times and then within 5 to 10 minutes, reconnect. The main office SBC is running in a VM on their Windows server and the remote office SBC is running on a micro PC running Linux.

John Rayfield, Jr.
Rayfield Communications

 

[pellisCMIT]

I did have another voip.ms outage at Monday 9:45 PM EST, I did have my SIP probe script checking each of the 3 ports that voip.ms offers and they continued to respond while the trunk was down.
42872/udp
5080/udp
5060/udp

I contacted my account rep on my previous outage that indicated it was only 3CX impacted and asked if this was something they could look into and was assured someone from 3CX would respond to the thread but still no response.

I've stopped the script as it's not impacting us any longer since we switched to Flowroute, but clearly a route is going down or something is bouncing in the path between the 3CX hosted instance and certain locations or providers, but it's hard to get someone to dig deeper into the issue. Good luck!

 

[JasonP]

Anyone find a solid solution to this? I am seeing similar on voip.ms and Flowroute all 3cx hosted.

 

[N_G]

These are 100% provider issues. Make a trunk on Twilio or Voxtelesys and you will see there are no issues an you can get support from Voxtelesys in case of issues.

In update 5 you can auto provision a Twilio SIP Trunk.

 

[JasonP]

Thanks for the suggestion I am going to give Twilio a try.

 

[N_G]

Great, if you continue to have a problem with any of those providers, you can open a support ticket. If there are no problems, then you can go back to your provider and say that there are no issues with Twilio so fix the problem yourselves.... Our hosted machines are hosted on Digital Ocean and have good connections to the internet btw.

 

[Rej]

I will add my own issue to this thread. For many months, I have brought these issues to VOIP.ms tech support, but they have told me the problem is not on their end.

I have 3CX systems deployed both on-premises and in the cloud (Azure and AWS), each using VOIP.ms trunks. Some systems also have additional trunks, but the issue occurs only with VOIP.ms. These systems include Canadian DIDs using Canadian VOIP.ms servers and US trunks using US VOIP.ms servers, and the same issues occur across both setups.

The major frustrating problems caused by this random unregistred issue is calls dropping unexpectedly while people are on the line, I have tried switching servers, but the issues persist.

 

[N_G]

So switch provider to twilio or voxtelesys