No Service after VPN tunnel interruption

[FBHLMike]

Hello,

We are currently experiecing an ongoing issue with our Yealink T46S phones and 3CX that we need some assistance in tracking down. When we upgrade Branch office Firewalls firmware; some of the phones at those branch offices will go to a "No Service" status while others will reconnect when the firewall and VPN tunnels are re-established. This same phenomenon has occured in the past after a power outage. This issue does not occur consistently however as out of 8 firewall updates completed in a night, only one office will have these complications.

Detail on our setup:

• 3CX locally hosted on dedicated server in corporate office with NIC team consisting of two interfaces with a single IP address.
• All branch offices connect to corporate office via Branch Office Virtual Interfaces (VPN Tunnels)
• All VPN routing is handled with OSPF
• All phones are provisoned to internal DNS name rather than IP address of PBX

When this issue occurs at branch office (ex: firewall reboots.) Firewall is down for maybe 2 to 5 minutes. Once firewall is back internet returns and VPN tunnels come back up. Some phones come back up without an issue. Other phones show "No Service". Unplugging those phones for 2 minutes usually gets them to register again, however some phones need to be reset to factory and reprovisioned.

In testing we can see "No Service" phones are fully able to ping the PBX server across the tunnel. And the traceroute from the phone shows traffic traversing the tunnel as expected. DNS is resolving correctly to the internal IP address of the 3CX server.

Capturing packets with Wireshark on the server, we can see the PBX recieving register requests from the phone and responding that authorization is required. However this cycle just repeats itself over and over and over and the phone never registers. The phone remains in a "register failed" state. See screenshot from Wireshark below.

This issue does not happen in a consistent and repeatable manner so it has become hard to track down. However Firewall FIrmware updates seem to the most reliable way of triggering it.

Any help and advice would be greatly appreciated in tracking down the cause of this issue!

 

[cobaltit]

Where is the DNS for these branch offices? I'm pretty sure in another thread this was identified as a DNS issue. Since you are using an internal DNS name if that doesn't resolve at the local site when the VPN is down that is probably the issue

 

[FBHLMike]

Where is the DNS for these branch offices? I'm pretty sure in another thread this was identified as a DNS issue. Since you are using an internal DNS name if that doesn't resolve at the local site when the VPN is down that is probably the issue

Thank you for your reply. We have two windows DNS servers. One is in our Corporate office and one is in a secondary office. Each location has a tunnel to each DNS server so it can use either for name lookups. Our DNS name is the same externally as it is internally. We just provision our phones Local LAN and have local A records.

It could definately be DNS, but I image we wouldnt be seeing the requests to register hitting the PBX if these devices were unable to resolve the DNS name correct? Also wouldnt it occur for all devices in the branch office rather than just a few?

 

[eddv123]

but I image we wouldnt be seeing the requests to register hitting the PBX if these devices were unable to resolve the DNS name correct?

I agree, I think from my experience DNS to be less likely, we have had cases at least in the case of SBC'd phones and the SBC itself, where register requests will not be even attempted if the device or devices cannot resolve the public IP to FQDN.

I have not personally experienced this issue but have worked with a lot of VPN'd 3CX sites and can tell you that in the case of DNS we will stick a dns record (3CX FQDN >> to local IP) on the device router and this works perfectly well.

We have had issues also where phones go into a "No Service" state but these occurrences have been down to the VPN (we think) going stale as it effects all phones (turning the traffic type from TCP and back to UDP refreshes the connection) and thus it continues on.

What I can say is that VPN's when single site to 3CX work very well but routing must be in place when multiple VPN's are being used. This obviously needs to be sorted on the network level first.

Yealink phones support ping and trace route in the phone interface under "Network >> Diagnostics" have you tried to run a trace route to the PBX from a non-working phone ? does it take the same route as one that is working ?

 

[YiannisH_3CX]

Hello @FBHLMike

From your description it looks like the phones can reach the PBX but the reply from the PBX does not reach the phones. If you open a REGISTER message sent by the device after the PBX requests for Authentication you will see that the phone is not sending the credentials. It is just repeating the REGISTER message.
When the issue reappears you will need to run a capture simultaneously on the PBX and on an affected phone.
Check if REGISTER messages are sent to the PBX and if the PBX receives them. Then check if the reply from the PBX reaches the phones.
You will also need to check the Contact IPs sent by the phone and where the PBX sends the reply. Also check if the ports are correct.

 

[FBHLMike]

If you open a REGISTER message sent by the device after the PBX requests for Authentication you will see that the phone is not sending the credentials.

I would like to check this. Where in the Packet would I be able to see or determine if credentials are being sent back? Assuming we do not see credentials that would seemgly confirm that the PBX is not reaching the phone in these situations, correct?

The IP addresses look good going back and forth, and so do the ports. Also traceroutes from the PBX to the effected phones show the same route as from the yealink phones web interface. Which essentilly just shows local gate, remote gate, destination since it is a VPN tunnel.

 

[YiannisH_3CX]

When a phones registers to the PBX, it first sends a Register message without a proxy-authorisation sip field. The PBX will then respond a 407 Proxy Authentication Required message.
The phone will then respond with a second Register message which should contain the proxy-authorisation sip field. The proxy-authorisation field contains a number of parameters necessary for the phone to authenticate to the PBX.
If you see the phone sending a Register message without this field over and over again then that probably means that the reply is not reaching the phone.