Inbound call audio Flowroute/GC VM

[easttek]

I am suddenly having an issue with choppy call audio on inbound calls only using a GC VM. This server has been deployed without issue for well over a year. This issue is on all phones in several locations, some via SBC some STUN. Phones (fanvil x series) and 3cx firmwares are up do date. VM size is appropriate for amount of extensions and codecs are set in correct priority in system/sip trunk/extension in accordance with flowroutes supported codecs. I do not see any severe usage of resources in 3cx or Google Cloud Console. Submitted a ticket with flowroute with a pcap and they said they are seeing degradation at the PBX/VM IP address. I restarted services, the VM instance, checked port forwarding at remote locations and tried several different codecs.

Any thoughts?

 

[YiannisH_3CX]

To be honest audio issues are very hard to troubleshoot so if you think the issue might be at the PBX site i would urge you to open a ticket with our support department so they can have a better look at it.
If you are familiar with wireshark you could troubleshoot the issue your self by capturing a call and exporting the 2 audio streams to see where the issue is.

 

[easttek]

Thank you for the response. I will open up a ticket. I have pulled a capture and the caller can hear the callee without distortion, the callee (system phones) have terrible audio incoming from the caller.

 

[luckystrikes]

I have the same issue. PBXs working fine for a long long time, I've found some workarounds so feel free to DM me. This is a HUGE issue. I will open a ticket with 3CX as well so I can help them identify the issue. It is totally random it seems and some PBXs are more problematic than others. I've moved the server to different regions and get more of the same.

 

[NickD_3CX]

Of course without checking a packet capture as my colleague suggested above, I can't be sure, but I would keep some reservations as to where the problem is being caused.

I will state something of statistical nature, we have seen quite a few threads pop up today regarding Flowroute and Call Quality.
Now, if you haven't touched your PBX since yesterday, odds are maybe it is not your PBX?

There is one way to tell for sure though, if you don't know your way around Wireshark, make a capture of an problematic call, then open a ticket with the 3CX Support Team and have them tell you their take on the matter.

 

[luckystrikes]

Hi Nick, yup I just opened & submitted the wireshark. The thing is, we haven't touched PBX since months ago besides updates and the issue has been reported mostly this week but with apparently customers experiencing but not reporting as early as last week and 2 weeks ago.

My workaround involved moving to AWS and testing and mostly resolved the issue for the customers however this morning it seemed to occur again although it subsided significantly (ie; every single call was a problem to 1 call this morning) Of course as this is across so many PBXs its hard to pinpoint.

The reason I thought it was PBX was when re-routing around the PBX, call quality is perfect. So I don't know if it is a 3CX issue, it may very well be a GCP issue. Although now I am less sure since it is happening again on AWS. I'm working on getting an AWS packet capture & dump and will update the ticket.

 

[NickD_3CX]

Soooooo, with my statistical thought above, plus the fact that it happens in 3CX PBXs in GCP and AWS, using the method of elimination, I know where I'm putting my money on...

Seriously though, our Support Department does a great job and usually is very detailed in explaining what the problem is and where, so I'd wait patiently a bit to get some hard data back.

 

[amir_]

Hi Nick, yup I just opened & submitted the wireshark. The thing is, we haven't touched PBX since months ago besides updates and the issue has been reported mostly this week but with apparently customers experiencing but not reporting as early as last week and 2 weeks ago.

My workaround involved moving to AWS and testing and mostly resolved the issue for the customers however this morning it seemed to occur again although it subsided significantly (ie; every single call was a problem to 1 call this morning) Of course as this is across so many PBXs its hard to pinpoint.

The reason I thought it was PBX was when re-routing around the PBX, call quality is perfect. So I don't know if it is a 3CX issue, it may very well be a GCP issue. Although now I am less sure since it is happening again on AWS. I'm working on getting an AWS packet capture & dump and will update the ticket.

Very interested in knowing what the conclusions are.

We have over 500 DIDs with Flowroute and have just started migrating from Cisco to 3CX. I've never had a significant issue with Flowroute but the few times I have had cases open - I'm left with the bad feeling that if I had to troubleshoot audio quality issues - those guys WOULD NOT be of assistance.

 

[easttek]

Update: Have not been able to chase this much further but am getting reports of improvements after switching to the lower bitrate G729 codec.

 

[luckystrikes]

Changing to AWS did help, the two additional reports which I mentioned earlier seem to be a fluke. I have continued to push flowroute for an answer. Hopefully someone else can advise, there seems to a larger thread going on

https://www.3cx.com/community/threads/inbound-call-quality-issues.81344/#post-375788

My guess it is a peering issue between FR/GCP and not 3CX related.