Solved Outbound calls being cancelled by phone across 2 different trunks

[CFMtech]

Hello Support,

Hoping you can point me in the right direction. Received a few reports that certain users are getting a fast busy signal when placing outbound calls to certain numbers. Here is what I know:

• 3CX Version, Professional Annual 16.0.5.612
• Server OS, Windows Server 2016
• Is the 3CX Server Hosted and where? On prem
• IP Phone Make/Model/Firmware: Grandstream GXP 2130 have tested on firmware 1.0.11.6 & 1.0.9.135
• Provisioning Method: Local / VPN / STUN / SBC Local LAN
• Trunk Provider or Gateway Make/Model Flowroute is trunk #1, Telnyx is trunk #2
• Has the Firewall Checker passed: YES
• Are custom Phone Templates being used: NO

This appears to only be happening to a select few individuals out of the 155 endpoints we have. I have tried with the default template, with custom templates, with the current firmware (1.0.11.6) and have rolled back the firmware on one of the affected phones to (1.0.9.135) and it is happening across all templates, firmwares and trunks.Have rebooted the phones, bounced the 3CX services and restarted the 3CX server with no change.

1. What could be happening here? I'm running out of things to check
2. Why would 3CX try to send the call out from both trunks? (flowroute and telnyx)

In the below the phone is 192.168.2.224 and the 3cx server is 192.168.2.49

 

[CFMtech]

Additional piece of info I just discovered, it appears all of the numbers that we are having issue with appear to be T-Mobile, yet our SIP provider is telling us they are not having issues passing off calls to the T-Mobile network. They're blaming the PBX since it looks like the CANCEL came from the handset.

I dont know if I believe that...

 

[leejor]

It does seem to indicate that the cancel was local

05/19/2020 2:29:09 PM - Leg L:2567.1[Extn:0224] is terminated: Cause: CANCEL from 192.168.2.224:5060

... and no SIP message from your provider, at least in the logs you posted. Was there a fast busy, or some other recording received, prompting the caller to hang up? If you set the logs to Verbose, is there any additional information? Is it possible that it is a caller ID issue since you seem to indicate that it is happening on two trunks (providers), to one destination (T-Mobile)? Perhaps your provider can check their logs of a failed call.

 

[CFMtech]

It does seem to indicate that the cancel was local

05/19/2020 2:29:09 PM - Leg L:2567.1[Extn:0224] is terminated: Cause: CANCEL from 192.168.2.224:5060

... and no SIP message from your provider, at least in the logs you posted. Was there a fast busy, or some other recording received, prompting the caller to hang up? If you set the logs to Verbose, is there any additional information? Is it possible that it is a caller ID issue since you seem to indicate that it is happening on two trunks (providers), to one destination (T-Mobile)? Perhaps your provider can check their logs of a failed call.

Thanks for the follow up leejor! Users report a fast busy. The provider is saying they see "session progress" response but then calls are canceled by the calling party. The above logs were with logging set to "medium" I've since changed it to verbose but its after hours so theres not much happening.

 

[leejor]

Your callers are hanging up because they get a fast busy. Obviously, based on the Activity Logs, there seems to be no SIP message returned, from the destination, be that T-Mobile, or some other intermediate carrier, indicating a problem, as one would normally expect.

I would ask your providers to see if they can duplicate the issue from another origination on their network. As I said earlier, the fact that it seems to be happening with calls over two providers, may mean they route the calls to T-Mobile in the same manner (via the same intermediate provider), or, the destination has a problem with something, such as the caller ID, you are sending

 

[CFMtech]

Thanks again @leejor The SIP trunking provider has reported to us that the upstream carrier has changed routes to the affected numbers, however we are still receiving a fast busy when attempting to call. I set the logging to verbose lastnight and capture this of the failed call:

Any thoughts? Appreciate it!

 

[leejor]

The SIP trunking provider has reported to us that the upstream carrier has changed routes to the affected numbers,

...and exactly, what does that mean? The calls no longer go to T-Mobile?

 

[CFMtech]

...and exactly, what does that mean? The calls no longer go to T-Mobile?

Honestly, your guess is as good as mine. Our SIP provider seems to acknowledge an issue with their upstream carrier and Tmobile numbers in our metro area, I believe the upstream carrier is attempting to route around whatever hardware/circuit is having issues. Perhaps there is some DNS propagation time or similar?

I have asked them to clarify, and sent them the above log snippet. As these things often do, the carrier is (at least partially) pointing the finger at the PBX and so far no one from the 3CX support team has chimed in with anything to send to the carrier to push back on them pointing at the PBX. Is there anyone from 3CX support that could chime in with some suggestions?

 

[leejor]

Our SIP provider seems to acknowledge an issue with their upstream carrier and Tmobile numbers in our metro area,

Does this mean that if they call one of the numbers, using their own network, they also get a fast busy?

 

[CFMtech]

Does this mean that if they call one of the numbers, using their own network, they also get a fast busy?

I've sent them 4 numbers that we are unable to reach, for one of them, they replied:

"On NUMBER we receive a "SIP/2.0 503 Service Unavailable" response from all available routes/carriers to this destination." This seems to have been the number that prompted their upstream carrier to change the route to the destination.

However on the other 3, they're saying they see a "session progress" response but then calls are canceled by the calling party, pointing to the PBX/handset saying we're cancelling the request from our end. I dont know enough about the inner workings of the SIP protocol to point out where we are missing a response from the provider or are sending an acknowledgement.

 

[leejor]

on the other 3, they're saying they see a "session progress" response but then calls are canceled by the calling party, pointing to the PBX/handset saying we're cancelling the request from our end.

...which may very well be true, if the caller receives fast busy, they're probably going to hang up, cancelling the call, I know I would. The perplexing part, is that there seems to be no reply passed along from from your provider, after the Session Progress, in those cases. If they are willing to test again with you, place a call, but don't hang up for a minute or so, see if the far end eventually does send some sort of Cancel message.

There is obviously something else going on beyond the testing/tracing capabilities of your provider.

 

[CFMtech]

@leejor I want to send a thank you your way, appreciate the assist through this.

I connected with some folks on reddit and spiceworks who were also encountering this issue, using the same provider. Once I realized it wasn't isolated to our systems we pushed hard on the provider, who in turn (at least this is what they're telling us, and I somewhat believe them) pushed back on the upstream carrier and somewhere over the long weekend (memorial day here in the states) it got corrected. I'm not yet ready to close up the ticket we have lodged with our provider but the team hasn't received any complaints yet today so I'm cautiously optimistic.

 

[leejor]

Ahhh... for the good old days when it was one telephone company, to deal with, end to end, and one complaint fixed a problem.

While competition is great, no one want to take ownership of a trouble, particularly when it passes from one provider to another. No one has enough time, or resources. It's easier to just point fingers, or say, "it's fine at my end....not my fault". Many customers want the least expensive (cheapest) service possible, and you get what you pay for, in most cases..