3CX and Skyetel 401 Unauthorized Registration Errors

[jerm324]

For the last week I have been having issues where occasionally one or more of my trunks would lose registration with a 401 unauthorized error at least once a day. After 5 minutes the trunks would re-register. This was on v18 update 7. After talking to the support staff at Skyetel and sending them some Wireshark packet captures they responded with this:

It has come to our attention that a recent 3CX update is breaking RFC and causing customers who use SIP Registration to have registration issues. Specifically, section 22.1 of RFC3261, deals with the mechanism for SIP registration authentication requests and challenges.

Here's a summarized version:
- When a User Agent Client (UAC) sends a request to a server, the server can challenge the UAC to provide credentials to prove its identity.
- The server does this by sending a response with a status code of 401 (Unauthorized) or 407 (Proxy Authentication Required).
- This challenge response includes a "WWW-Authenticate" or "Proxy-Authenticate" header field, respectively. These headers contain a challenge value that the UAC must use to produce a valid response.
- The UAC responds to the challenge by resending the request with an "Authorization" or "Proxy-Authorization" header field, which contains the credentials.
- The credentials are typically a combination of username, realm, nonce (a one-time token from the server), and other values, which are used to create a "digest" using a specific algorithm (usually MD5).
- The server then verifies the digest. If it's valid, the server processes the request; if not, it can challenge the UAC again or reject the request.

Unfortunately, 3CX is no longer accepting the 401 as a valid response, and as such it is breaking the standardized way to handle SIP Registration.

We have reached out to them for clarification, and will update you when we hear more.

They mentioned a "recent update" although I had not yet updated to v18 Update 8 when this started. I have now installed this update but am not sure it will resolve the problem. Can someone with 3CX clarify what is going on?

The error I was getting looked similar to issues described in these threads albeit with a different SIP provider:

https://www.3cx.com/community/threads/authentication-error-because-of-using-same-nonce-by-3cx.61476/
https://www.3cx.com/community/threads/did-logic-call-or-registration-to-did-logic-has-failed.68724/

I can send in the Wireshark packet captures if needed.

 

[SweetAction]

https://www.3cx.com/community/threads/call-or-registration-failed-401-unauthorized.122610/

People running update 5 are reporting this issue, so it's really unlikely to be something 3CX did in a recent update. If it starts recently for multiple users and people with 1 year old PBXs are reporting the problem...

 

[bigdessert]

Yes this is something for sure on the Skyetel side. Unfortunately it is something that is going to cause us to stop doing business with them as they seem to be unwilling to correct this issue. We are already starting to make our move to another provider which is extremely painful but must be done.

 

[jerm324]

https://www.3cx.com/community/threads/call-or-registration-failed-401-unauthorized.122610/

People running update 5 are reporting this issue, so it's really unlikely to be something 3CX did in a recent update. If it starts recently for multiple users and people with 1 year old PBXs are reporting the problem...

The link you provided does not work for me?

 

[SweetAction]

The link you provided does not work for me?

Sorry, that link is Partner only. My bad on that.

But this post from that thread is good to know

So we looked into this matter

A. Nothing changed in regards to our voip provider code with 3CX over the past few updates. We have no other reports from other providers.
B. Our behaviour is correct when receiving a SIP error 401.
C. We have seen similar issues with Skytel over the past year. These were duly reported to Skyetel.
D. We dropped them down several levels but we did not want to completely remove them as we are genuinely interested in working with them.
E. The agreement with supported providers is that if they have something they want to ask us or they dont agree with they can reach out to our interop team. Our interop team received nothing.
F. Rather then public finger pointing they are to escalate tickets to our team for review.

It seems Skyetel has a new owner who is interested to work with 3CX but some changes need to be made to the platform in terms of reliability. We are going to pull them temporarily from 3CX but we hope that the new owner will change things at Skyetel and so we hope to re introduce them as soon as we are convinced of their new commitment to 3CX. We will keep you posted.

 

[jerm324]

Sorry, that link is Partner only. My bad on that.

Ok gotcha thanks. Were there any other insights in that thread that I should be aware of?

 

[SweetAction]

Ok gotcha thanks. Were there any other insights in that thread that I should be aware of?

I would also keep this in mind

@bigdessert Thank you for the heads up. To be honest we have since spoken to the new owner and I think there is a genuine will to fix this from the part of Skyetel so lets give them a chance. Maybe it was a one off. From our side we are fine to hit the reset button so to speak.

 

[jerm324]

I would also keep this in mind

Thanks a bunch I appreciate it!

 

[jerm324]

For anyone following this and doesn't have access to the partner forum here is the latest response I got from Skyetel:

We greatly appreciate your patience in this matter. We have heard directly from 3CX and they have confirmed with us an update on their PBX is causing this behavior. Both our Engineering Teams are working side by side to resolve the Registration issues. The goal if all goes well is to be able to provide resolution to this issue today. Our Support Team will provide more updates as they are made available to us.

Both sides pointing fingers on blame still but hopefully resolved soon either way.

 

[jerm324]

If anyone is following this, Skyetel told me they did an update last Wednesday, and for a few days we had no issues, but come Sunday (yesterday) the issue returned, also experienced it again today. About done with these guys if they don't sort this out.

 

[bigdessert]

We also were sitting good late last week when they said they had implemented a fix but it is back as of yesterday again. I have submitted two ticket replies to them(once yesterday and once today) with absolutely no replies.

 

[jerm324]

I sent in a message early yesterday and finally got a response this morning 10 minutes ago. All they said was "Thanks for reaching out. Our Engineering is engaged in reviewing your request. We will follow up once more information is available." Meanwhile I'm getting quotes from other providers.

 

[bigdessert]

Funny, I just got the exact same reply just now....they aren't trying very hard to correct this and we are also working on going to another provider as this is unacceptable.

 

[Skyetel]

Our apologies for the delay in replying to this. There's a lot of background to cover, but the bottom line is that we made errors in both our initial troubleshooting of the root cause of the issues and in how we communicated them to our customers. I want to take a moment to outline what transpired and why our initial response about 3CX users was inaccurate.

What Happened

This is a brief technical summary – if you would like more specific details, we are happy to provide them via a ticket. Posting about the idiosyncrasies of the root process pausing children workers so it can retrieve information from a database and load it into a hashtable here is overkill .

Last week, our network encountered a near-catastrophic routing issue that caused our entire SIP infrastructure (IP Auth and SIP Registration) to behave irregularly. This affected our customers in various ways. IP Authentication customers experienced higher than usual PDD and call drops, while SIP Registration customers faced authentication issues, as reported by 3CX users.

Regrettably, it took our team more than a day to pinpoint the root cause. All our monitoring indicators, including CPU, RAM, Network Ports, and UDP Buffers, showed nominal utilization. The root cause was eventually identified as a higher-than-acceptable lag in the import time for Skyetel’s routing tables into our core router’s RAM. This delay was due to a team member over-allocating routing resources without adequate consultation with our engineers.

Compounding the issue, we couldn't reverse those routing resources promptly because of core limitations in how our network manages specific PSTN resources. The only solution was to significantly increase the routing resources on our network during production hours, accepting the resultant service disruptions.

We managed to restore our IP Authentication network to full health on Wednesday, September 20th, and deployed a permanent resolution on 9/26. Our SIP Registration network, with its unique constraints, needed more time to address the authentication issues. We largely mitigated these issues by last Wednesday, but a full resolution is set for release first thing tomorrow morning.

Why Our Response Was Inaccurate

The reason is less technical and more cultural.

During the period our network faced challenges, all our technical resources were dedicated to identifying the core problems and finding solutions. We failed to establish a clear communication channel between our engineers and our front-line customer support representatives. This gap allowed conjecture to be mistaken for fact, leading to inaccurate company communications. To be clear – our initial assessment was incorrect, and the SIP 401 code we received from 3CX was RFC compliant. This was our bad.

On a personal note, I (Chris) did not maintain clear communication either. I was immersed in the thick of it with my team, frustrated, and was much more focused on fixing the technical challenges than clearly communicating. For that, I am sorry. I also apologize for my delay in just posting this. I have been working very closely with my technical teams getting these issues permanently resolved, and this is the first opportunity I have had to reflect and provide transparency into what happened.

 

[jerm324]

Our apologies for the delay in replying to this. There's a lot of background to cover, but the bottom line is that we made errors in both our initial troubleshooting of the root cause of the issues and in how we communicated them to our customers. I want to take a moment to outline what transpired and why our initial response about 3CX users was inaccurate...

Thank you for the response and explanation. I will keep an eye on the error logs and let you know if we encounter more issues.

 

[Skyetel]

Hi All - we pushed out the last updates for SIP Registration this morning as promised. From what we can see, all is working as it should now. If you have any additional issues, please do not hesitate to reach out to our support department for resolution. Thank you again for your patience and understanding

 

[N_G]

Thanks for the heads up @Skyetel / Chris. Glad the matter has been addressed and thank you for the openness.

 

[KyleKSIP]

@Skyetel @Nick Galea I noticed Skyetel is not listed in the supported list for 3CX SMB. I assume it is related to this thread... Is there any update or timeline on when they may potentially be added?

 

[N_G]

No Skyetel is not listed as their service is not compatible. Additional requirements are needed to be compatible with the multi tenant edition of 3CX - namely rinstance - this allows us to quickly identify the SIP trunk tenant.