occasional one-way audio routed/no-nat

[dandenson]

I've got some hosted 3cx instances with VPN connectivity. I'm running pings across this and see zero packet loss.

Every now and then, a call will come in (external > ring group > phone, g.711 or g.729 tested) and when picked up, the remote side (cell phone) get's no audio or gets some light background distortion sounds.

Dial again and it works.

again, there is no NAT here. phone>router>L2TP_tunnel>hosted_router>3cxPBX all routed no NAT. T48G phone provisioned to use the LAN address and everything seems to work just fine except this occasional one way audio.

I would expect that without NAT in the way, one way audio should only happen if there is an ongoing issue with the routes. This being intermittent makes me think there's a 3CX issue (15.5R6).


so, looking at the activity log I see this:
Call to T:Extn:000@[Dev:sip:[email protected]:5060] from L:41.1[Queue:800] failed, cause: Cause: 488 Not Acceptable Here/INVITE from 10.50.0.199:5060
[CM503003]: Call(C:41): Call to <sip:[email protected]:5060> has failed; Cause: 488 Not Acceptable Here/INVITE from 10.50.0.199:5060

10.50.0.199 is the phone I'm testing against. Now, the last one-way-audio call was at about 3:20 today and those logs are at 10am so I don't know if it's related but a 488 doesn't makes sense because the phone actually does ring and I at least get audio in one direction.

 

[cobaltit]

Is the remote side (cell phone) the 3CX app or are we talking PSTN? Audio issues are 99% network firewall so the first thing to do is make sure on the 3CX firewall you have the full RTP range open (9000-10999) and that the firewall test pass). Internally (3CX to endpoint) you need to make sure 7000-8499 is open. Since you say you have a VPN with no NAT I'm assuming this is already the case but just wanted to have a good base.

Since you are talking about no audio the pings are irrelevant. No audio in one direction is a network issue or a configuration issue, Network quality in the form of packet loss doesn't come into play here. Packet loss also wouldn't induce distortion sounds. 488 Not Acceptable here means it doesn't like something in the invite. This could possibly be a codec issue. Please confirm we are dealing with a supported 3CX setup. So this would be V15.5 Update 6, firmware on .55a and stock 3CX template.

 

[dandenson]

remote cell doesn't use app, this is a PSTN in to SIP trunk.

the 3CX firewall is what is built during the PBX express installer, and it's firewall is very basic.
the remote side firewall isn't doing NAT at all. It's a mikrotik with a route to the PBX over an L2TP tunnel

The 3CX instance can ping the LAN IP of the phone and a PC on the phone's subnet (10.50.0.0/24) can ping the 3CX instance confirming that there is no NAT in the way.

yeahlink T48G on 3CX current firmware (66.83.0.55) and 15.5U6

MTU is 1450 across the layer 2, 1500 on the AWS lan.

As far as the 488, I think that was me testing codecs specifically ilbc so that's probably a non-issue and hasn't shown up again.

The private ip to private ip pings are my indicator that connectivity is good.

 

[cobaltit]

Since you haven't specifically stated it can you confirm the firewall checker passes in 3CX? If this is an older instance I don't believe it will auto-correct the RTP range that was expanded in Update 6 so you'd have to do that manually.

 

[dandenson]

yes, the firewall checker passes. again, please note that this is effectively 'LAN' and avoids the firewall completely.

 

[cobaltit]

You are talking about one way audio with an external call so the firewall on 3CX matters although if it's outbound audio only that's probably not it. It doesn't sound like this setup should have this issue and the intermittent nature is going to make it hard to track down. Have you run the firewall checker after the upgrade to Update 6? If there are no reports of internal calls having one way audio I still thing it's your 3CX firewall. The only way to be sure is be able to PCAP the call in progress on the PBX to figure out if audio is making it from the phone to the PBX so you know which side to look at further.

 

[dandenson]

correct, outbound leg is the audio that either isn't making it or has some flaw. I do get some 'sound' but it's barely perceptible and is kind of robotic or just a few little garbles.

I'm watching the firewall and what's weird is that I'm still seeing ~65kbps and I'm seeing that leave the 3cx instance on net-top.

 

[leejor]

Is this happening only when calling the one mobile number? Just the one user extension doing this sort of thing? Any other issues with outgoing calls, at all? Do all calls route over the same trunk group? Have you tried repeatedly placing a direct call to the same mobile number? If you can replicate, a direct call to the mobile number, i would contact your provider. It could be a bad span of trunks between them and the mobile provider. Have you looked at the Activity Log right after this has happened just to confirm routing and see if there are any error messages?

 

[dandenson]

leejor, it's actually happening when I call in from the cell(pstn). Call hits a queue which includes 2 phones (T48G, T48S). I've been testing primarily against the T48G because it's on my desk.

I don't know that outgoing calls ever have the issue. I tested a few calls this morning and it works fine.

The activity log doesn't show anything unusual on the calls that had one way audio. I do see the message in this thread:
https://www.3cx.com/community/threads/strange-messages-after-update-to-15-5-sp2.52267/

that looks like some yealink firmware issue though.

 

[Christopher Reagoso]

Strongly suggest installing 3CX SBC and not tunneling through your VPN if possible for a number of reasons, chiefly being you probably cannot prioritize traffic you're carrying in your tunnel.

But if you SBC to the 3CX, it'll all be on 5090 and you can prioritize.

That'll just dispense with a ton of troubleshooting.

Next- Pcap in front of the tele and see if you hear drops.

Next 2 - Pcap at the phone system to see if you hear the same drops.

If you hear the same drops at both locations, it might not be you.

 

[leejor]

Sorry, misread the original post.... If the issue happens only with one phone, then obviously that is the suspect. If it happens with one make/model of phone, then it is probably firmware related although it could be an option misconfiguration.

Replace with another phone, one of the same make, to begin with (after the whole factory reset/re provision thing), if the problem goes away, then if not a firmware related, it may simply be hardware related (bad set).

 

[dandenson]

Christopher, I'm happy to use the SBC *BUT* I need to do it over the VPN because that VPN is what's providing seamless connectivity (ie, hot failover to cell without dropping sessions). As far as I can tell, the SBC requires the FQDN and uses that to connect to. If I could connect the SBC to the tunnel address that would be great.

Also, I know all the IP addresses at play here so I can prioritize traffic based on that, the ports don't even matter.

leejor, it's not just one phone, I can get the same on the T48S and a T29P also.

 

[cobaltit]

@dandenson

The phones also use the FQDN no? And that traffic finds its way across the VPN so using the SBC should be drop-in.

 

[dandenson]

cobaltit,
STUN - phones use FQDN
SBC - phones use FQDN *BUT* they also use the proxy. The proxy has the FQDN in DNS and points that to the LAN IP entered into the SBC's configuration.

I don't believe it works to put the LAN IP in for the fqdn. Maybe if I hard coded lan ip : fqdn in /etc/resolve.conf on the SBC I could hack it.

 

[cobaltit]

I think it through me off when you said tunnel address. You don't want to connect to the tunnel address unless the VPN is on your 3CX instance (which is unsupported). You just want your FQDN to resolve to the internal IP address which is routed via VPN. So either putting the FQDN in the host file on the SBC or pointing the SBC to DNS servers which resolve the FQDN to the internal IP should do the trick.

 

[Robertos]

Yes would need to see packer captures with sip sdp information and subsequent rtp streams from 3cx would be a good start

 

[dandenson]

I've traced down the scenario. It's the *first* call after a period without calls. If I place a call on the phone to 'prime' it, then I can do any number of inbound calls without an issue. If I let it sit for 10 minutes, maybe 30% of the first inbound call will have one way audio.

Tracing that back, I can see that the first ping across the tunnel will have a high latency and sometimes drop the first packet. So I think this is lost data in the invite.

I'm sending a constant ping across the tunnel to keep it alive and I'll test more today.

Unfortunately, I don't have an option for TCP on these yealink phones with the stock provisioning which I would imagine would cure this.