Solved Intermittent failed outgoing calls - v16

[Frick]

I am posting to see if anyone else is having a similar issue. After upgrading to v16, we are getting intermittent failed outgoing calls (cross numberous 3cx pbx instances). When you try to call the phone just shows it is trying to make the call you get silence. There is no ringing or anything. After 32 seconds you finally get an error and the 3CX event log says:

"Call or Registration to xxxxxxxxxxx@(Ln.10001@xxxxxx) has failed. 0.0.0.0 replied: 408 Request Timeout; internal"

If the number is tried again later, it may or may not work. It is completely intermittent.
One thing we noticed is that while we have all outgoing calls to use 11 digit dialing (prepend a 1 to 10 digit calls), all the call logs show the failed calls as only 10 digits. All successful calls have the 11 digits in the call log.

We have had this happen with two SIP providers and have also tested/verified our network is working normally, leading us to believe it is related to 3CX v16.

Has anyone else had similar issues?

 

[Nicholas James Pionek]

Yes, starting around middle of last week I have seen 7 of our sites with this issue on v16. Rebooting the PBX seems to allow calls to resume going outbound but the problem could crop up again at any time. Did extensive outbound call testing over the weekend, put over 2,000 calls through and about every 6 hours or so it would start to fail again until you rebooted. This has become a headache already this week with more and more sites calling me saying they are not able to call out. I get the same internal request timeout message as Frick did.

 

[Frick]

I am somewhat relieved for my own sanity that someone else is having the same issue. Troubleshooting this has not been easy.

Here is an example of what we are seeing. All calls are set to call out using 11 digit dialing via an outgoing rule. The failed calls are being reported as 10 digits. A call to the same number fails multiple times (showing 10 digits) then will finally complete (showing 11 digits).

 

[JohnS_3CX]

@Frick can you post a picture of the outbound rule that is meant to process the failed calls in the screenshot?

 

[Frick]

Here is our 10 digit call rule:


Here is our 11 digit:


The only other rule we have is one for 911 calls.

 

[Nicholas James Pionek]

These are the same rules we have as well, only the 11 digit rule we do not specify it having to start with 1.@Frick

 

[JohnS_3CX]

Hi Frick, I think you will need to spend some time on WireShark for this one.

When you see the logs missing the prepended digit "1", it means that the call was not successful in using the trunk and there was no point in including the digit since the trunk did not respond.

When the call succeeds the logs include the "1" prepended digit after the event completes, the system has to write the log as a last step.

Use tshark if it is not possible to replicate easily so you can catch it.

 

[Frick]

Hi John, thank you for that info. That clears up the 10/11 digit confusion for us. We already have a ticket open with 3CX and our SIP trunk provider.

On the 3CX side, it has been determined that the SIP carrier is not responding to the call request. Our SIP Trunk provider says they never see the request at all on their end. We are pretty confident it is not a network/routing issue based on testing connections while the issue is occurring, and that fact many of our 3CX instances all use the same networks, but dont have the issue at the same time.

We have also tried using a different SIP Trunk provider and experience the issue there also.

 

[Frick]

Just to follow up on this, we have narrowed it down to be an issue with Tenlyx as the SIP provider, and specifically the DNS resolution of their main registration URL intermittently not resolving correctly. We have put in the IP address and have not had the issue since.

 

[JohnS_3CX]

Thanks for updating the thread, I'm glad you found a workaround for now and hopefully the DNS issue will be resolved.