STUN phones with WAN failover

[Matt (Ignite Solutions)]

Hi team,

I'm back with another re-occurance of the same problem that I've written about earlier:
https://www.3cx.com/community/threads/yealink-t58-407-proxy-authentication-required.58528

The same client had the same situation this week, and I think I've narrowed it down in my head as to what the root cause is. I'd like to run my logic by you all to see if you agree with my reasoning, and if you have any idea on how to solve it permanently.

Re-cap on the scenario:

• Cloud 3CX PBX running in Azure, deployed by PBXExpress.
• Most of the time, it works fine. Passes all firewall checks, kept up to date, etc.
• 19 extensions at the primary office site. A single Yealink IP Phone (reception extension), 18 Windows 10 softphone apps and 18 iPhone softphone apps (each staff member is registered on both their desktop and their iPhone).
• ALL phone provisioned via remote STUN. This is important.
• Meraki MX64 firewall, with two WAN internet links (different providers) for failover. This is important.

As I mentioned, most time, things work fine. But a WAN failover event induces headaches that last for days. Wednesday night, the primary internet link dropped out for 30 mins at midnight. We've been struggling with 3CX issues ever since.

What I think is happening:

• The phones connect to the PBX via STUN. As part of this, both the PBX and the phone learn their public IP address and port.
• When a WAN failover occurs, the Meraki unit seamlessly remaps existing "flows" from the Primary interface onto the Secondary. It does this without dropping the connections.
• Hence, the STUN connection is still in place.
• When the PBX needs to place a call to the phone, it tries to connect to the public IP and port that it knows for that phone. This is the Primary WAN's IP address, which is down.
• Phone can talk to PBX for some things (outbound calls work, for instance, because the PBX's IP doesn't change), but the PBX can't talk back to the phones. Things like BLF's take a long time to update. Call transfer takes a long time to complete, etc.

This holds with my findings in the previous thread, you can see the packet captures where the PBX is trying to talk back to the client on a different public IP address than the one the client is talking out on. The penny didn't drop for me at the time, but it has now.

My thinking here is that the Meraki is being "too smart". If it were to simply drop those long-running STUN connections when it failed over the WAN, the internal phones would simply re-connect to the PBX. This would trigger the public IP address mapping for the phone to be updated. Everything would work.

But because the Meraki is moving the in-progress connections onto the Secondary, neither the phones nor the PBX realise that the public IP has changed.

I see four ways to solve this:

1. Change the way the 3CX PBX behaves, such that it will only hold a STUN connection open for, say, 5 minutes before it closes it. This would force the phone to re-establish the connection out, and give an opportunity for the system to recognise the change in public IP. But it sort of defeats the purpose of a long-running STUN connection.
2. Change the Meraki to a different firewall that's much dumber. When it fails over, it will just drop all current connections and allow them to be re-established. This is basically what happened to the softphone on your mobile device when it changes from wifi to 4G as you walk out of the house. Not a great solution, because the Meraki is otherwise a perfect bit of kit for this client's needs (also, it was an expensive investment for the client).
3. Change the internet link to something like a private MPLS network that can have multiple uplinks into the core cloud, but a single public IP egress point inside the carrier network.

Has anyone else come across a similar situation before when using dual-WAN failover at a site that has STUN-provisioned phones? How did you solve it?

Cheers,
Matt

 

[YiannisH_3CX]

Hello @Matt (Ignite Solutions)

I am not familiar with Meraki firewalls so i cannot tell you what is going on there but perhaps i can provide some info on how the PBX works in this scenario.

The 3CX clients should be configured to use the tunnel so they do not use STUN. In the scenario that the connection is dropped the PBX will reset the TCP connection and it will try to re-establish it. Again not sure what the Meraki is doing in the event of failover.

The Yealink will use STUN before every action (Register, Invite etc) to know its IP address and contact the PBX. But the STUN is not a constant connection. The phone will use STUN to find it's IP before a registration attempt and will send the Register message. There is no constant connection however with the PBX. The next Registration attempt from the phone will occur 60 seconds after the previous attempt. In those 60 seconds the PBX has no idea if the phone is there or not.
You can test this by unplugging a phone right after it re-registers to the PBX. The phone will remain visible in the management console for the duration of the Registration period (usually 120 seconds).

You could start a capture on the Yealink device and on the PBX and wait for a registration attempt from the phone. Once that happens you can look at the STUN request from the phone and what reply it is getting. Then compare what the phone is sending in the Register and what the PBX is seeing.