3CX DNS SRV Resolution Issues

[TecMate]

I have multiple instances of 3CX running all in different environments and different SIP Trunk providers all with what appears to be the same problem.
Outgoing calls will fail intermittently with the following error:

SIP Server/Call Manager ID: 12294​

Call or Registration to 1800#######@(Ln.10000@Skyetel) has failed. 0.0.0.0 replied: 408 Request Timeout; internal​

All instances running 3CX Pro 16.0.2.910
2 instances are Google Cloud Hosting deployed via 3CX PBX Express.
1 instance is on-prem Intel NUC Core i7 running Windows 10 Pro 1903
SIP Trunks are Skyetel (USA) and Breeze Connect (AUS) both are using the 3CX Certified Templates with no changes to their settings.

To me this looks like a DNS resolution problem. 0.0.0.0 is obviously not my SIP provider's IP address...

The host name for Skyetel is: transport-tcp.3cx.skyetel.com which doesn't resolve to anything...
So I checked with Skyetel engineers and they told me that host name was suggested by 3CX to force use of TCP (preferred by 3CX) instead of UDP for SIP connections (not RTP)
and 3CX looks for DNS SRV records for _sip._tcp.3cx.skyetel.com
and this returns:

Non-authoritative answer:​

_sip._tcp.3cx.skyetel.com service = 20 50 5060 ca1.skyetel.com.​

_sip._tcp.3cx.skyetel.com service = 10 50 5060 va1.skyetel.com.​

_sip._tcp.3cx.skyetel.com service = 30 50 5060 or1.skyetel.com.​

Great! Except for there is still a problem given the failed outbound calls and the error above.

I have tried changing each instances DNS servers to Google 8.8.8.8 and 8.8.4.4 and CloudFlare 1.1.1.1 and 1.0.0.1
That did not fix the problem.

I have also tried creating a custom SIP Trunk and adding Skyetel's static IP addresses into the host name field. Then setting this custom SIP Trunk for outbound calls.
This does resolve the issue. However I want to know:
Why is the 3CX Certified template is not working?
How does 3CX specifically handle DNS resolution (SRV, A records, order of operations, etc.)

 

[YiannisH_3CX]

Hello @DanTron

The 3CX template includes the registrar that resolves to the SRV records of the provider. You want to use SRV records so you can have failover between those records.
So even if we assume that the DNS is not working the template has nothing to do with it. The template is there to provide you with a pre-configured setup so you do not have to manually adjust all settings.

To be able to determine what is not working and why some calls are failing you will need to do some troubleshooting.
The first step is to switch the logging level of the PBX to verbose and restart your services you have more details being logged. (remember to turn if off when you are finished troubleshooting as it is resource intensive).
When you replicate the issue you can look into the activity log for the call and see why it failed.
You can also run tshark on the server so you can catch a failing call. That will show you if the call is reaching the provider or not and if you are getting a error back from the provider.

 

[TecMate]

@YiannisH_3CX you advising me to just stick to the template and use the SRV records. As I explained, that's what I am using. But I'm still receiving these errors:

Call or Registration to 18003284880@(Ln.10000@Skyetel) has failed. 0.0.0.0 replied: 408 Request Timeout; internal ​

Again using the logging, what does this error tell you?

@cobaltit on the forum pointed me to this documentation.
https://www.3cx.com/docs/sip-trunk-dns-record-resolution/

Which helps but what I don't understand is where/why the DNS resolution process is breaking down.

 

[YiannisH_3CX]

The error you are receiving is very generic and could mean a number of things. The call may not be leaving the system for some reason or you may be getting a unavailable response from the provider. As i mentioned in my previous reply you need to dive deeper into the logs to find the cause.

 

[Eduard Lavilla]

I have 3 Instances in Azure and the 3 are giving this intermittent error only outbound call I was having the error with Helifox sip trunk and I change to Skyetel bevuase I though that it was helifox fault but still the errors

Now yesterday I create a new account in Azure and create a new Instance and same error and it look like is a network issue not DNS becuase my first Sip trunk was only IP

this is been happening for a month now and my client are about to kill me
Help

 

[StefanW]

before we go further

1. Update to v16 U3
2. After Update check, if the firewall checker passes in full

then we will proceed.

 

[Eduard Lavilla]

before we go further

1. Update to v16 U3
2. After Update check, if the firewall checker passes in full

then we will proceed.

I have this version 16.0.676 and I click update and it say that is up to date
and the firewall test say everything is ok

how can I upgrade to 16 U3?

 

[StefanW]

the message was more for Dan (post owner) he is on 3CX Pro 16.0.2.910

 

[YiannisH_3CX]

Hello @Eduard Lavilla

How often do calls fail and are you using the default registrar and the default template for Skyetel?

 

[Eduard Lavilla]

Hello @Eduard Lavilla

How often do calls fail and are you using the default registrar and the default template for Skyetel?

yes we use template from skyetel ( but I had another sip provider everything IP ) and the same error

The call is random 3 calls go out and the 4th fail or maybe the first one fail and the 3 work fine

in the capture I see that the error is that it looks that the SIP provider do not answerback but the issue is that the package never get to the sip provide

 

[Eduard Lavilla]

This is the capture when the error happen 50.17.48.216 is Skyetel IP so it does resolve fine

 

[YiannisH_3CX]

So the IP is resolved correctly and the Invite is sent to the correct IP but the provider is saying that the Invite never arrives to them?
Since you have a capture can you send it to me in a p.m.? Since you cannot upload it directly you need to provide me a download link.

 

[TecMate]

before we go further

1. Update to v16 U3
2. After Update check, if the firewall checker passes in full

then we will proceed.

Stefan

All instances were upgraded to U3 about a month or so ago and the new "Auto Discovery" box is ticked on the Registrar Hostname for the main SIP Trunk. Also all instances are passing the firewall test with no problems. I have also checked and my firewall rules are set to allow UDP and TCP on port 5060.

I'm also trying to get a packet capture of a failed call but it's intermittent and I can't leave a capture running for days at a time...

I've cranked up the Activity Log detail to Verbose on one of my instances that's running on a Core i7 to hopefully get a detailed log of what's happening. I'll report back once I have more detailed logs.

 

[YiannisH_3CX]

@DanTron

You can use tshark and leave it running until the issue is replicated.

Install Tshark and start capture doing the below:
apt-get install tshark
mkdir pcap
chmod +7777 pcap
tshark -i any -b filesize:10240 -w pcap/capture.pcap

This will create 10 mb files until the issue is replicated and will not stress the system will keeping one large pcap file.

 

[Tamer_Great]

We having the same issue and Skyetel is requesting to contact 3cx to update their template

Here is the reply from Skyetel:
The bug in 3CX has to do with SRV records and is impacting all carriers who use it. We strongly recommend you report your issue in that forum as well - the more data they have the better

 

[Skyetel]

Hi All,

I figured we'd chime in here to provide some useful insight. Here's what we have observed with our users have put in tickets about this problem:

1. Its intermittent - it happens frustratingly infrequently and irregularly
2. We do not receive the INVITE packet. However, because this is intermittent, we get the calls immediately preceding and immediately following the call that was not delivered to us. Because it doesn't resolve, Wireshark does not appear to pick up the failed call (It acts like 3CX never sent it)
3. It appears changing DNS servers to public DNS servers like 1.1.1.1 or 8.8.8.8 does not make any difference (though we recommend using these servers anyway)
4. Multiple customers have confirmed that this impacts other providers too - not just us.

We'll continue providing feedback and insight as we get it Thanks to 3CX for their help on this with us - they've been helpful with us working behind the scenes.

 

[YiannisH_3CX]

@Skyetel

Thank you for your valuable input on this.
I would strongly recommend to share the data you have with us so we can look into them. Since the transport of choice is TCP the Invite will not be sent if the TCP connection to your servers fail. Did you check if the TCP connection was established?
The data we have collected so far indicate that the IP address of the SRV record is resolved correctly and the Invite is sent to the correct IP.
So if you have collected data that show something different we would love to see them.

 

[YiannisH_3CX]

@Tamer_Great

Can you share some info about your system and how it is setup?

• 3CX Version, e.g. Standard Annual 16.0.2.910
• Server OS, e.g. Debian 9 / Windows Server 2012 R2 / Raspberry Pi
• Is the 3CX Server Hosted and where?
• Has the Firewall Checker passed: YES / NO

Thank you

 

[Skyetel]

@YiannisH_3CX my apologies - some of our users are saying that this is happening with other providers, but we do not have specific data to support that. So I cannot say for sure that is true and I should not have implied as such.

 

[YiannisH_3CX]

To those who have a Skyetel trunk and are face the issue with the 408 Timeout try switching the trunk to UDP and see if this resolves your issue.
To do so edit your Skyetel trunk and navigate to the Options tab.
Select UDP under the Transport Protocol.


Let us know if this resolves your issue.