iOS and Android apps on cellular: tunnel rebuilds every ~8.5s

[MCJ]

Self hosted v20 U9 (build 995), no SBC. Every mobile app user on cellular gets audio in 1 to 6 second bursts, a beep, "Recovering", then they hang up. T-Mobile and AT&T, iOS and Android, current builds. Same app and phone on district Wi-Fi is flawless.

WAN edge packet capture of the failing calls (port 5090, TCP control plus UDP media) shows a repeating 8.5 second cycle: the app opens a UDP flow, sends about 3.3 seconds of audio, then abandons that port completely while the PBX keeps streaming at 50 packets per second into it for another 4.6 to 4.9 seconds. At 8.5 seconds the app opens a new UDP port and a new TCP connection and starts over. The app sends the RST or FIN itself, the PBX never closes first and answers each new port within about 100ms.

For reference: on Wi-Fi one TCP connection held 324 seconds and one UDP flow held 329 seconds with zero gaps. Cellular but idle, one port held 90 seconds or more. The churn only happens while a call is active.

Firewall side is clean: 21 to 22 sessions per phone across 20 to 22 source ports per window, all accepted, zero denies, destination NAT only with no source port rewrite, no DoS policies, IPS clear.

Control test: a laptop tethered to that same phone over cellular, using the 3CX web client, 55 second call, single UDP media flow, 2,747 packets up and 2,729 down, continuous, no audio problems.

Questions:

1. Known issue on carriers using 464XLAT and carrier grade NAT, where the app drops its tunnel media flow after a few seconds?
2. What makes the app abandon the flow after 3.3 seconds and rebuild the tunnel at 8.5, on cellular only? Is there a client timer?
3. Can the mobile apps be forced to tunnel over TCP, or is KEEPALIVE_TIME_UDP=30 it?

 

[Claas Felix Beyersdorf]

Sounds like DPI / Flood Protection Problem.

5090 TCP is only for the SIP traffic.
5090 UDP for the RTP traffic.

So for me it sounds like something is blocking the RTP traffic and the descriptions sounds like some kind of flood protection.
Each directions requires 50 Pakets per second as 20ms Paketization Time. So some firewall might block this as false positive for flood.

While the phones are using WIFI are they inside the same network as the PBX or are they in the WIFI inside Home office?

For me it sounds like a problem with the firewall. Which can be tested as described. Try to use the Phones inside an WiFI on another location if the 3CX phone shows the same Problems its not the carrier.

 

[MCJ]

Two results, as promised. Neither supports the carrier theory, and the second one surprised me.

External Wi-Fi test. Same phone, same app, home connection on an unrelated ISP, nothing shared with our network and no carrier involved. It fails identically: short audio bursts, beep, Recovering. Capture from that call: 10 tunnel TCP connections in 111 seconds, each about 6 seconds, and every one closed by the app itself with a RST. Zero retransmissions on any of them. The PBX answered each new port in under 100ms and kept streaming 50 packets per second into the abandoned port for 4.6 to 4.9 seconds after the app stopped sending. In the same window the app's own HTTPS connections to that same PBX, over that same home network, stayed up for the whole 109 seconds without a single reconnect.

So it fails on two mobile carriers and on an unrelated ISP, iOS and Android. It works on our internal network, where the client reaches the PBX without NAT, and the web client works over the same external paths. The common factor in every failure is a NAT'd client arriving over the internet, and the side closing healthy connections each time is the app. That also retires the 464XLAT and carrier-grade NAT theory from my first post. So, disregard that part.

Forced TCP test. I said I would block UDP 5090 at the edge so the app had no choice but the TCP tunnel, then call and see whether the media path is what it cannot tolerate. It did not stabilise. With UDP 5090 blocked the app never moved media to TCP. It kept retrying UDP for the whole call, about a hundred denied flows across four different source ports in two minutes, while the TCP 5090 session stayed up in 20 to 25 second stretches carrying roughly 1 KB/s. That is keepalive, not audio. A tunnel call is about 200 KB every 20 seconds. Same symptom: bursts, beep, Recovering.

So this build has no TCP media fallback. TCP 5090 is the control channel and the audio has to go over UDP, which narrows the failure to one thing: the UDP media flow is abandoned mid-call while the PBX is still sending at 50 packets per second into it.

Two questions, for anyone from 3CX reading:

Does the v20 U9 mobile client support TCP-only media at all, or is TCP 5090 control-only by design in this build? Every capture I have says control only.

What makes the client stop sending on a UDP flow that is still receiving 50 packets per second from the PBX? Something that treats the flow as dead, either a timer or a health check that wants inbound media first, would explain the pattern, because at the packet level the path is healthy when it is abandoned.