Phones via VPN to 3CX server - Delayed Audio

[pjr]

Hi all,

A bit of background on this topic. We provide our clients with VoIP systems and host them in house with all sites phones connecting to their hosted dedicated 3CX server via hardware VPN tunnels using a mix of 10Mb Fibre lines or dedicated ADSL circuits.

Prior to 3CX we used an alternate VoIP system that had feature limitations but was rock solid with call quality (the actual SIP gateway element of the previous solution was a dedicated hardware device - Ingate SIPerator firewall).

We changed to 3CX last year after a successful 3CX trial with our internal system and now have several 3CX based installs in use for clients. On each site they have all mentioned experiencing intermittent issues with delay on VoIP calls. This is not something we experience in house on our system (but we are on the same LAN as our system) or expected to encounter for our client systems.

We have looked at various possible causes from the comms in use on the clients site, to the SIP trunk provider in use, but in each case nothing appears to be the cause and in any case it's the same comms and SIP trunk provider/LAN topology that were used in our previous non 3CX implementations that didnt experience these issues.

We then began to think that it must be 3CX that's the cause and that the software SIP/Media gateway is adding additional overhead to the media processing and this is causing the delay, but if this was the case I would expect the delay to happen on each and every call (which it doesn't)?

My question is - is anyone else out there using 3CX with remote phones connected via VPN and experienced/troubleshooted similar issues?

My experience of troubleshooting VoIP issues of this nature is limited but we have taken call recordings of "delayed" calls from the server and cannot hear any delays, yet we can hear the participants in the call commenting on the delay in the actual call audio itself.

We have also taken wireshark captures on the server and can see the 2 legs of the call in wireshark but I've not used wireshark to this level of depth before and am not sure which legs of the conversations shown will make up the audio received by each party - also, will the capture taken on the server necessarily reflect any delays experienced at the client site?

I presume to get a true picture I also need a capture taken off a mirrored port onsite which can then be compared to the server based trace to see if the delay is being added by the VPN?

It's a massively frustrating problem to get to the bottom of and it's not helped by it's intermittent nature. I could understand if poor comms were to blame but I would expect this would manifest itself in the way of dropped rather than delayed audio? We also used the same comms with the different system before 3CX and didnt experience these issues.

Can anyone offer any advice/assistance with regards for what I should be looking at in traces/3CX logs?

We have an engineer going to a client site tomorrow to do some local captures so hopefully that will provide a clearer picture.

I am championing 3CX within our company but I will not be able to progress much further if I can't prove this problem is not caused by 3CX.

Thanks

Paul

 

[leejor]

Is the delay extension to extension or extension out to a trunk (Voip provider or Gateway)? Do both ends experience it? Are the extensions set for "PBX delivers audio" ? Can you set up an extension, at the remote location, for testing, that doesn't use VPN, see if the delay is still there? Are all of the codecs set the same?

 

[pjr]

Internal calls - ie ext to ext are fine
The delay appears to be only on calls that route via a VoIP Provider (which has to be set to PBX Delivers Audio)
Both ends notice the delay in audio
We will be setting up a non VPN based ext on site tomorrow to test
The codec in use is G711A

 

[leejor]

We will be setting up a non VPN based ext on site tomorrow to test

That's about the only way to determine if the encryption is adding the delay. How about a call from a remote extension (VPN) all to an extension at the same location as the 3CX (non VPN) ? That would eliminate the VoIP provider from the equation.

 

[pjr]

We will be setting up a non VPN based ext on site tomorrow to test

That's about the only way to determine if the encryption is adding the delay. How about a call from a remote extension (VPN) all to an extension at the same location as the 3CX (non VPN) ? That would eliminate the VoIP provider from the equation.

Thanks Lee - we haven't tried that yet but as it's intermittent it's difficult to tell if we tried it and the problem wasn't there if it was the VoIP provider or just that the problem wasn't present at all during those test calls.

Worth a go though!

Paul

 

[leejor]

Intermittent is always much harder to diagnose. Is it always both incoming and outgoing, or just one or the other? Are the far end calls to a long distance location or to a phone in the same city? It's funny how "I always get echo/noise/other voices, on my calls", and when you ask people to keep track of the calls, it becomes... "oh, yeah, I guess its always when I call..." Since some calls don't have the delay and all calls go over the VPN, the VPN may not be the culprit after all. It could very well be be something your VoIP provider is doing in routing/handling certain calls.

 

[pjr]

Looking at the traces that were taken onsite yesterday it would appear that the delay between one party speaking and the other responding is present on the leg between the voip provider and the 3CX server. When looking at the capture on the server, there is a small additional delay in the stream from the 3CX server to the phone (which I would expect and attribute to media processing overhead), but this gap is present in the stream captured from the switch on site but is identical to the same stream when captured on the server - this leads me to believe the issue is not caused by the VPN aspect adding any further delay. The delay seems to be at source.

Also the calls captured yesterday whilst onsite seem to indicate the delay is on inbound calls only. We are checking with the client to verify if this is always the case or not.

We need to ascertain now if the delay we can see in the wireshark capture is caused by the voip provider or 3CX itself.

Any tips?

Paul

 

[leejor]

You might want to see if your VoIP provider is willing to get involved (do some monitoring) while you try some test calls.

 

[SY]

Hi Paul,

Could you please send me those wireshark captures?

Thanks

 

[pjr]

Hi Stepan - I've sent a PM

 

[FBW]

I don't know if this info will be helpful for you but here's our experience with calls over VPN.
We've been using this setup on our own 3CX PBX for a year now without any problem.

We do use the GSM codec which reduces the needed bandwith. This has some advantages if your bandwith is limited. And we've had no reduction in line quality due to this.

We have 2 remote sites connecting via VPN to the PBX and it works fine. Even if we call between the 2 remote sites there's no loss in quality.

I must say that we recently succeeded in imrpoving reliability of our connections (even direct ones from the PBX to the SIP provider) using reserved bandwith. Unfortunately not all routers/firewalls support this feature.
Recently we tried the same for a customer using a WatchGuard. And that one unfortunately only supports traffic prioritization.
Our own Lancom routers on the other hand DO support this reservation. We tested with 4 simultanious calls and running a speedtest.net at the same time... the line didn't even budge!

Diederik
http://www.fbw.be