Audio "drops"(pause) for user(s) Part 4

[TLCtech]

• 3CX Version, Enterprise Annual 18.0 Update 1 (Build 237)
• Server OS, Debian GNU/Linux 10 (buster),4.19.0-18-cloud-amd64
• 3cx hosted
• Provisioning Method: Cloud?
• Trunk Provider: Call Centric
• Has the Firewall Checker passed: YES
• Are custom Phone Templates being used: NO

Even after uprading our router to a Ubiquity Dream Machine Pro, bathed each room in the office with wifi, and after moving our PBX off site and upgrading to enterprise, we STILL are having this issue. Im very frustrated and am at my witt's end. What more? Its apparently only one extension now.

On a call with anyone, in office, working at home, or clients on their own phones, one extension drops audio every 60-90 seconds. Ive listened to calls. She experiences drops in audio that last 2-3 seconds where the person she is talking to drops out completely. They also experience a drop in audio for 2-3 seconds, and then a garbled bit as it "catches up". Ive listened in on calls. This seems like an issue locally still. I dont hear any drop of the person our agent is speaking to, but the agent hears it, every 60-90 sec. But the customer and I both hear when the agent drops.

I ran verbose logging and captured a call. I also recorded a call. For some reason we cant record every call we want to either. Listening in, I was unable to record any call and my agent had the same problem.

The only thing Im seeing in Wireshark is this, repeated every few moments.

5695 2021-10-29 14:56:32.600980 IP ADDRESSES TCP 66 [TCP Retransmission] 62051 → 5001 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
5696 2021-10-29 14:56:32.601010 IP ADDRESSES TCP 54 5001 → 62051 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

7259 2021-10-29 14:56:37.674473 IP ADDRESSES TCP 66 51138 → 5001 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
7260 2021-10-29 14:56:37.674491 IP ADDRESSES TCP 54 5001 → 51138 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

7338 2021-10-29 14:56:37.931809 IP ADDRESSES TCP 66 50583 → 5001 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
7339 2021-10-29 14:56:37.931838 IP ADDRESSES TCP 54 5001 → 50583 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

7436 2021-10-29 14:56:38.227506 IP ADDRESSES TCP 66 [TCP Retransmission] 51138 → 5001 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
7437 2021-10-29 14:56:38.227540 IP ADDRESSES TCP 54 5001 → 51138 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

7514 2021-10-29 14:56:38.476972 IP ADDRESSES TCP 66 [TCP Retransmission] 50583 → 5001 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
7515 2021-10-29 14:56:38.476999 IP ADDRESSES TCP 54 5001 → 50583 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

7607 2021-10-29 14:56:38.781722 IP ADDRESSES TCP 66 [TCP Retransmission] 51138 → 5001 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
7608 2021-10-29 14:56:38.781754 IP ADDRESSES TCP 54 5001 → 51138 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

This is beyond frustrating. We have been dealing with this for over a year. What is causing this and why does it continue to plague us? I was able to save 3cx from the axe last month, but at this point, my clients are willing to pay Vonage prices if this goes away.

 

[cobaltit]

Since it happens across so many devices and seems to persist, and it's only one extension, is it only on inbound calls on a particular DID?

 

[TLCtech]

So that's where this is slightly different.

So far, only one extention is reporting it. And before it was only experienced by people calling in outside the office. And it was less predictable. Now, it happens regularly, with one extension, and both outside calls and calls within the office, experience it on both sides.

Listening in, I was not hearing the outside audio breaking up, but our agent was. And we both heard the agent cut out and back in. Every 60-90s

 

[cobaltit]

So I'd try an extension to extension call, or even an echo test from the affected extension and see if you can isolate it to your environment or CallCentric.

 

[TLCtech]

So we can talk ex to ex, and it happens then, but only with that one extension. And it happens when shes in the office on her computer and the other extension is outside the office. Her desk is in a poor reception area, or I would have asked her to stay on cellular and take calls via the app. And I have taken her wifi antenna and moved it to the center of her office, closer to the AP than everyone else not having issues. Ive also replaced that antenna. The only other variable is that she takes calls mostly from the browser interface and not using the windows app. Ive asked that she switch to see if the problem persists.

 

[TLCtech]

But, in one form or another, this issue has plagued everyone in the office no matter what I have done. Ive completely upended the entire network and still, the issue persists. The only major variable now is the software. At least thats how the client feels, and Im beginning to feel this way.

 

[TLCtech]

Oh and as for cell phone reception, she is able to take and recieve calls just fine on cellular, but the lte can be average at best. Takes wifi calls just fine.

 

[SweetAction]

[TCP Retransmission] usually means that you have packet loss (so it was retransmitted because no ack). It jives with garbled audio, out of sync audio, audio "catching up", etc.

Try a continuous ping from the affected machine to the 3CX server. Are you dropping packets?
Also one to the AP, to the router, and to some site externally. Depending on what drops you will learn it's AP, it inside the network, it's internet specific, etc.

Any chance on hardwiring in the PC instead of WiFi? You mention a Dream Machine so I assume UniFi WiFi APs and well, ubnt isn't known for great bug free firmware. (I'm sure there is a firmware upgrade to fix that bug but it will introduce a new bug.)

 

[TLCtech]

This isnt new at all unfortunately. We replace OUR ENTIRE NETWORK and we still have this issue. Im at my wit's end. We cannot hard wire in our current building. The patch panel is garbage and none of the ethernet ports in the walls work at all. None. Zero. They will not pass traffic through. The cost to replace the panel is more than they want to spend when they are moving in a year.

What I may have to do is run some ethernet to that room and set up a switch. The AP has LOS with the wireless adapters in this room but still we have dropped and garbled audio. At this point, My client is seriously wondering if its the phone system, and its hard to argue against it. Its been a full year of this issue. When you replace everything in the network 3x, and even move the pbx off site, in the cloud, and this is still an issue, its not a good look. And to have a 3cx partner come in would be almost the same as replacing that patch panel. Why doesn't it just work?

 

[10m]

Have you tried using a desktop IP phone? I know you said you use wireless but when I had similar issues like this one, I changed the endpoint. I found that the user's computer was not working very well. I also asked the users to try the 3CX app on their mobile devices. And just throwing this in there, this problem started after the v18 upgrade.

 

[TLCtech]

No, this issue has been going on a year.

Because we cant hard wire, we cant use hard phones. Because no one else is reporting it, I know its likely with the user. BUt its the SAME ISSUE WE HAVE HAD FOR A YEAR. And with this client, no news isnt usually good news. Its them using their cell phones to make calls and then getting upset when they cant xfer to a line because its not in our system. But I digress.

I keep asking the user to use the app on their phone, but its been hard to get them to do so as their flow usually hinges on being able to work hands free. But its something I push to troubleshoot. The problem comes when we have been through all of this before. So they are super tired of dealing with it and throwing more time and money at it. Im not trying to dismiss your suggestions, but it really has been exhausting.

 

[SweetAction]

I'm trying to help you here, but I need some answers. I think your next step should be to determine if it's network or if it's 3CX specifically.

Try a continuous ping from the affected machine to the 3CX server. Are you dropping packets?
Also one to the AP, to the router, and to some site externally. Depending on what drops you will learn it's AP, it inside the network, it's internet specific, etc.

As shown here: https://fastgood.cheap/ you got to pick. Either put in the legwork to troubleshoot, pay someone else to, or have a poor experience.

 

[TLCtech]

I understand. Trust me, Ive been doing a lot of troubleshooting on this issue FOR A YEAR and so far I havent found anything that definitively says what the issue is. Ive tried pings, Ive tried seeing if packets were dropping using wireshark, and Ive come up with nothing. You can see my post history. Im at a loss. I am pretty sure its packet loss, but I cant actually see it. I can watch it happen. I remoted into the station through 3cx and the video just lags, the audio cuts out. The connection from the AP to the router is fine. The wifi in the office is the best its ever been since the overhaul. This is the only complaint anyone has are the phones.

I had the user try from their phone app and it was the same issue. Its not the machine. And everything on the router side says this shouldn't be happening. Im going to try and hard wire her desktop, but at this point, unless others are just not reporting any issue, it almost feels like the extension is the issue. Its not the computer. Her phone app has the same issue. And hard wiring it means more construction they dont want.


We paid for the service in hopes that moving it off site would help. And it didnt solve the issue. Neither did overhauling the network. Not fully. Im tired and I have no answers.

 

[TLCtech]

Its gotta be location based I guess. Seeing as shes using the app next to her PC...

 

[SweetAction]

So if you've already tried the series of continuous pings mentioned (to AP, to Gateway, to 3CX, and to the internet) - what were the results? Did you drop to all 4 destinations at the same time? Or only specific ones?

My post here still stands: https://www.3cx.com/community/threads/audio-drops-pause-for-user-s-part-4.85378/post-401267

Since you've said:

So that's where this is slightly different.

So far, only one extention is reporting it. And before it was only experienced by people calling in outside the office. And it was less predictable. Now, it happens regularly, with one extension, and both outside calls and calls within the office, experience it on both sides.

Listening in, I was not hearing the outside audio breaking up, but our agent was. And we both heard the agent cut out and back in. Every 60-90s

That tells me almost 100% you will drop all 4 at the same time, because the Agent's machine is likely losing connection to the network. The fact that a listen in shows only the agent disappearing and re-appearing shows not an issue with 3CX or the SIP trunk.

I get no one wants to spend money to hardwire in this machine. How much money are they spending to have you troubleshoot and in lost calls? Just drape a 300ft cord across the floor for testing to the nearest switch. Or not.

 

[TLCtech]

So if you've already tried the series of continuous pings mentioned (to AP, to Gateway, to 3CX, and to the internet) - what were the results? Did you drop to all 4 destinations at the same time? Or only specific ones?

My post here still stands: https://www.3cx.com/community/threads/audio-drops-pause-for-user-s-part-4.85378/post-401267

Since you've said:

That tells me almost 100% you will drop all 4 at the same time, because the Agent's machine is likely losing connection to the network. The fact that a listen in shows only the agent disappearing and re-appearing shows not an issue with 3CX or the SIP trunk.

I get no one wants to spend money to hardwire in this machine. How much money are they spending to have you troubleshoot and in lost calls? Just drape a 300ft cord across the floor for testing to the nearest switch. Or not.

Thats what I was able to talk them into. The plan is to bypass the patch panel and terminate the lines going to that room. If they test well, we can use the outlet. If not, we will just run it the best we can to that office and install a switch there.

The company that would actually come out to fix or bypass the panel wanted $2k to do so. They just didnt want to invest in something that large that they couldnt take with them next fall. If bypassing the panel works, then every office should already be ready to wire, which also means they could finally get hard phones. SO we will see.

Mods, any way we can keep this thread up through at least the 20th? I know they have me work on other stuff and I dont get back to these, and they close, but I dont want that this time. I wont be able to get to that until the 20th.

 

[ChrisC_3CX]

Mods, any way we can keep this thread up through at least the 20th? I know they have me work on other stuff and I dont get back to these, and they close, but I dont want that this time. I wont be able to get to that until the 20th.

I don't see why not. Though do note that threads may auto-lock due to inactivity, if that does happen just PM me so that I can unlock it when you're ready to post a reply.

 

[TLCtech]

Ahh ok thanks. So we ended up replacing the patch panel (company wanted $2k to do that) and which allowed for HW for any office. Most still use wifi, but this user is a desktop. They havent been in this week yet so we are still waiting for that.

 

[ChrisC_3CX]

Thanks for coming back on this! Keep us posted!

 

[TLCtech]

Hard line seems to fix the issue with one agent. I think our wifi issues are causing other issues, so Im optimising that. For now, I think I have a handle on this. Ill come back if something else comes up