Solved Yealink T58 - 407 Proxy Authentication Required

[Matt (Ignite Solutions)]

Hi there,

I'm hoping someone can help me with a persistent issue I'm having with a customer's 3CX environment. We provisioned the 3CX server through PBXExpress, and it's running in Azure. The customer is mostly softphones (Windows 10 and IOS), and they work fine. The receptionist has a Yealink T58 with the EXP50 extension module to manage calls.

Initially we had a LOT of trouble with the T58. Inbound calls would work fine, but if you tried to call out the phone would time out and eventually show an error on screen : "Invite request no response". On the 3CX side, there's no trace of the call in the activity log.

We played around with it at the time for about a week, and suddenly it started working!!! So we left it alone. Tonight we've had to relocate a bunch of the networking equipment at the customer's site. And now, the Yealink is back to it's old tricks.

The phone is provisioning correctly, and it's showing up as registered/online.

However, I've run a Wireshark trace, and I don't think it's actually registering.

• I see a "REQUEST: REGISTER sip xxx.xxx.xxx.xxx" message come in to the PBX from the handset.
• Then immediately after, I see a "Status: 407 Proxy Authentication Required" response from the PBX back to the handset

I suspect the handset isn't understanding the 407 message, so it just sits there and waits till it times out.

I don't understand why the PBX is requesting proxy authentication though? The handset is provisioned as a remote STUN (they all are, no SBC, no tunnel). Handset config is fine, and hasn't changed between the past 3 weeks or working and tonight where it's broken.

I've rebooted the PBX, defaulted and re-provisioned the phone, etc. There are no blacklisted IP addresses.

The only thing I can think of is that during tonight's network changes, there was a period where the firewall lost it's primary internet connection and failed over to the 4G. This would have made the IP phone register from what looked like a different public IP address. Does the PBX "remember" the IP address that a phone used to register? Is that why it thinks this one is now coming through a proxy?

Any thoughts at all?

Cheers,
Matt

 

[YiannisH_3CX]

Hello @Matt (Ignite Solutions)

Please note that the 407 Proxy authentication is normal and the PBX will use it to authenticate an endpoint. The same happens when the PBX tries to register to a provider. In that scenario the provider is the server and the PBX the client so the provider requests the 407 Proxy Authentication.

The question is does the message come back to the phone and is the phone replying correctly?
Do you see a reply from the phone containing the authentication info requested?
I would recommend running a capture from both sides at the same time, one on the PBX and other through the phones web interface. Make sure that the response comes back to the phone and the phone replies correctly.

 

[Matt (Ignite Solutions)]

Hi @YiannisH_3CX,

Thanks for helping out.

OK - so the 407 Proxy Authentication is nothing to worry about - that's good. I've been staring at this so long I've got myself tied up in knots and I'm chasing ghosts.

I've got a capture from the PBX side, but I don't have one from the phone's interface. My tech has left the site now (it's after 10pm here), so we'll try to capture one tomorrow from both sides.

Is there anything that I should be looking for in the PBX side trace? Looking at the trace here, I can see:

• Request: INFO sip:[email protected]:5060
• Status: 481 Call/Transaction Does Not Exist

During this process, the PBX doesn't show any calls in progress, and there's no call record logged. It's as if the PBX isn't seeing a request from the phone at all.

Cheers,
Matt

 

[YiannisH_3CX]

The first thing to check is that the registration is done correctly. Also make sure that SIP ALG is turned off on the remote site firewall. As the phone is provisioned is using STUN you will see that the phone has a SIP port defined under the extension settings / Phone provisioning tab. If the phone is provisioned with the default template and supported 3CX firmware you should see the registration requests arriving at the PBX from the port. If you see a random port then this could mean the firewall at the remote site has SIP ALG enabled or port remapping enabled.

Are there any other IP phones provisioned in the remote site?

SIP Info messages are used by the PBX as a DTMF method so i do think this is related to the issue.
Run the captures on both sides and let us know of the results.

 

[Matt (Ignite Solutions)]

OK - I've got a call trace from both the phone and the PBX now.

From the phone side, I can see:

• Phone -> PBX: "Request: REGISTER sip:customername.3cx.com.au:5060"
• PBX -> Phone: "Status: 200 OK"
• Phone -> PBX: "Request: INFO sip:extension@pbxIP:5060"
• PBX -> Phone: "Status: 200 OK"
• Phone -> PBX: "Request: INVITE sip:[email protected]:5060"
• Phone -> PBX: "Request: INVITE sip:[email protected]:5060"
• PBX -> Phone: "Status: 407 Proxy Authentication Required"
• Phone -> PBX: "Request: ACK sip:[email protected]:5060"
• Phone -> PBX: "Request: INVITE sip:[email protected]:5060"
• Phone -> PBX: "Request: INVITE sip:[email protected]:5060"
• Phone -> PBX: "Request: INVITE sip:[email protected]:5060"
• Phone -> PBX: "Request: INVITE sip:[email protected]:5060"
• Phone -> PBX: "Request: INVITE sip:[email protected]:5060"
• Phone -> PBX: "Request: INVITE sip:[email protected]:5060"

From the PBX Side, I can see at the same time:

• PBX -> Phone: "Request: INFO sip:extension@siteIP:61444"
• Phone -> PBX: "Status: 200 OK"
• Phone -> PBX: "Request: REGISTER sip:customername.3cx.com.au:5060"
• PBX -> Phone: "Status: 407 Proxy Authentication Required"
• Phone -> PBX: "Request: REGISTER sip:customername.3cx.com.au:5060"
• PBX -> Phone: "Status: 200 OK"
• Phone -> PBX: "Request: INFO sip:extension@pbxIP:5060"
• PBX -> Phone: "Status: 200 OK"
• Phone -> PBX: "Request: INVITE sip:[email protected]:5060"
• PBX -> Phone: "Status: 407 Proxy Authentication Required"
• Phone -> PBX: "Request: ACK sip:[email protected]:5060"

So that flow looks good to me.

There is one thing that IS weird and sticks out like a sore thumb though.

Remember how I mentioned that earlier in the night, the firewall had failed over to it's 4G connection because we were re-patching everything? The SiteIP in the "from the PBX side" trace is the 4G IP address, not the primary fibre link IP Address.

Even looking inside the packetson the phone trace, where the Yealink phone is talking to the PBX, the packet source IP is the internal (LAN) IP of the phone, but the SIP header and body all refer to the external IP on the 4G link:

Via: SIP/2.0/UDP 4GIPADDRESS:61444;branch=longidfield;rport
From: "User's Name" <sip:[email protected]:5060>;tag=xxxxxxx
To: "User's Name" <sip:[email protected]:5060>
Call-ID: )_xxxxxxxxxx@LANIPofPhone
CSeq: 94 REGISTER
Contact: <sip:extension@4GIPADDRESS:61444>
and so on​

My theory is that, for some reason, the Yealink phone has established the STUN connection using the 4G link, and something means that it remembers that original external/public IP address. The firewall has since failed back to the primary WAN IP Address, but the phone is still talking to the PBX as if it's the 4G IP Address. The PBX is trying to talk back to the phone using the 4G public IP Address. But because the firewall has failed back, it's blocking any communication from the PBX back to the phone.

SO.

How do I make the phone forget that it used to route out via the 4G interface on the firewall, and instead make it detect that it's got to go out via the primary WAN interface instead? I've already rebooted both the phone and the PBX, and I've defaulted the phone and completely re-provisioned it.

I'm going to try to reboot the firewall now, see if that helps anything. The only think that I can think of is that the firewall hasn't cleared out it's routing table properly, and it's still routing packets from the phone out the 4G link instead of our the primacy WAN. IT's a long shot, but right now I'll take anything.

Thoughts?

Cheers,
Matt

 

[Matt (Ignite Solutions)]

Thanks for your continued help here @YiannisH_3CX

The first thing to check is that the registration is done correctly. Also make sure that SIP ALG is turned off on the remote site firewall. As the phone is provisioned is using STUN you will see that the phone has a SIP port defined under the extension settings / Phone provisioning tab. If the phone is provisioned with the default template and supported 3CX firmware you should see the registration requests arriving at the PBX from the port. If you see a random port then this could mean the firewall at the remote site has SIP ALG enabled or port remapping enabled.

Are there any other IP phones provisioned in the remote site?

SIP Info messages are used by the PBX as a DTMF method so i do think this is related to the issue.
Run the captures on both sides and let us know of the results.

Registration looks to be correct and successful. The phone is showing up in the Phones section of the 3CX dashboard, and the extension light turns green in the Extensions section.

SIP ALG is turned off on both the site and the cloud firewalls. The PBX firewall checker completes with all green passes. The site firewall is a Meraki unit, which does not do SIP ALG at all (https://documentation.meraki.com/zG...Cisco_Meraki:_F.A.Q._and_Troubleshooting_Tips) .

The phone is provisioned with the default template, and is running the latest supported firmware installed from the PBX.

I think it may be to do with port remapping. If I'm looking in the right spots, the phone is provisioned to use port 5065 ("Local SIP Port of Phone"), but it looks like the requests are arriving at the PBX with a src port of 61444.

No other IP phones at the site. However, I do have a test Yealink T56S that I deployed to the site initially as a proof of concept. It worked faultlessly, connecting back into my 3CX environment. There are lots of softphones at the site, both Windows 10 desktop apps and IOS apps.

Let me know if you'd like any other details from the network captures.

Cheers,
Matt

 

[YiannisH_3CX]

Try making a call from the Yealink while running a capture on the PBX. Once it fails search the capture for STUN requests as the server acts as the STUN server for the phone. See which IP the STUN requests comes from.

 

[Matt (Ignite Solutions)]

It was the bloody firewall.

Specifically, a Meraki MX device. Configured with a primary WAN and a secondary 4G failover connection.

Earlier in the evening we moved a lot of the cabling around at the site - they're renovating their office and moving the IT cabinet down the other end of the building. Everything was unplugged, relocated, re-patched and plugged back in.

The patching for the primary WAN wasn't right though, so the Meraki fired up and immediately switched over to it's 4G link like a good little firewall.

After 30 mins, we noticed that the primary WAN wasn't lit up, found the problem, and plugged it in. The Meraki detected it and failed everything back from the 4G to the fibre link.

Easy-peasy, right?

Wrong.

Meraki equipment has this concept of "flows". So my browser requesting an image from a web server is a flow. So is a long-running SIP conversation.

When a fail-over event occurs, the device has no choice but to fail over every flow because the primary is down. If this causes calls to drop and need to be re-established, so be it, there's no alternative.

When a fail-back event occurs, however, the device does not force over any existing "flows". In-progress flows will stay on the interface that they were on until they finish. So if I'm half-way through a big download when the primary comes back, my download will continue to finish on the 4G. Any new connections will be directed out the primary as a new flow. Eventually as flows finish up, all the traffic will seamlessly and nicely migrate back to using the primary without forcing anything nasty like call dropouts.

However, there's two Bad Things with this plan:
1) SIP STUN "flows" don't ever end. Because the phone needs to keep the long-running connection to the PBX open through the firewall, they keep chatting to each other. This serves to keep the "flow" open and running. So the Meraki leaves it where it is, on the 4G. Which wouldn't be so bad, except...

2) Once the Primary is back up, the Meraki refuses to accept inbound UDP on the 4G, I think. I suspect that it moves the NAT rules back to the primary WAN, so when the packets arrive on the 4G interface, it just drops them. So we end up in a situation where the phone talks to the PBX, and the Meraki directs it out the 4G link because it's part of an already-established flow. But when the PBX talks back to the phone over the 4G connection, the Meraki ignores it and drops the packets.

NB - this is all hidden under the covers. If you look at the routing table in the Meraki interface, it shows the default route as out the primary WAN. But that doesn't tell the whole truth!

To resolve, I VPNed in to the site, logged into the 4G modem connected to the Meraki, and rebooted it. This dropped the ethernet link to the 4G modem, causing the Meraki to see the 4G link as offline. Because it now had no choice, it forced everything back onto the primary WAN. The 4G link came straight back up, but the flows were all homed onto the Primary now.

I can now log into the Yealink and make calls to my desk phone and my mobile perfectly.

Stupid bloody firewall.

So now I have a lovely firewall that can do automated failover to 4G, which is essential for a site with IP phones. However when it fails back, an effort to avoid a 2 second call dropout results in a never-ending denial-of-service against the primary reception phone. You can bet I'll be raising a job with Meraki about this. I'll post back with the results.

Many thanks to @YiannisH_3CX for your help - it was looking at the wireshark traces from the opposing sides that made the penny drop.

1AM - time for bed now.

Cheers,
Matt

 

[YiannisH_3CX]

Glad to see you found the issue and thank you for updating the thread with your solution. I am sure others will find this helpful and will save them a lot of time.