3CX Call Quality Bad

[ycit.ash]

I have searched something exactly what I am experiencing but have not found it.

I have all required ports open and all ports are statically mapped when running firewall check most ports pass with no error. The issue is that a lot of random ports in the 9000-10999 range fail. Most seem to pass with the normal 'done' in green, but a bunch fail with the 'full cone test failure' error. All of the previous tests pass.

I have tried packet sniffing to find the issue, but the output is incomplete not showing all the ports in the range. I am using pfSense 2.4.4p3. The pfSense was a VM, but I have switched it to a physical box and I seem to be having slightly better performance on the physical box, but the 3CX FW Test still fails random ports in the 9000-10999 range.

I am having terrible call quality to boot...

 

[craigreilly]

need more info -
what is the firewall especially...

 

[ycit.ash]

craigreilly,


The firewall is a physical pfSense running 2.4.4-p3. I have followed the guidelines for setting up the ports as per 3CX documentation. I do get passes on everything except random ports in the 9000-10999 range. See a partial output below,

resolving 'stun-us.3cx.com'... done
resolving 'stun2.3cx.com'... done
resolving 'stun3.3cx.com'... done
resolving 'sip-alg-detector.3cx.com'... done
testing 3CX SIP Server... done
stopping service... done
detecting SIP ALG... not detected
testing port 5060... done
starting service... done
testing 3CX Tunneling Proxy... done
stopping service... done
testing port 5090... done
starting service... done
testing 3CX Media Server... failed (How to resolve?)
stopping service... done
testing ports [9000..9398]... failed (How to resolve?)
testing port 9000... done
testing port 9002... done
testing port 9004... done
testing port 9006... done
testing port 9008... done
testing port 9010... done
testing port 9012... done
testing port 9014... done
testing port 9016... done
testing port 9018... done
testing port 9020... full cone test failed (How to resolve?)
testing port 9022... full cone test failed (How to resolve?)
testing port 9024... full cone test failed (How to resolve?)
testing port 9026... full cone test failed (How to resolve?)
testing port 9028... done
testing port 9030... full cone test failed (How to resolve?)
testing port 9032... done
testing port 9034... done
testing port 9036... done
testing port 9038... full cone test failed (How to resolve?)
testing port 9040... done
testing port 9042... done
testing port 9044... done
testing port 9046... done
testing port 9048... done
testing port 9050... done
testing port 9052... done
testing port 9054... done
testing port 9056... done
testing port 9058... done
testing port 9060... done
testing port 9062... done
testing port 9064... done
testing port 9066... done
testing port 9068... full cone test failed (How to resolve?)
testing port 9070... done
testing port 9072... done
testing port 9074... full cone test failed (How to resolve?)
testing port 9076... done
testing port 9078... done
testing port 9080... done
testing port 9082... full cone test failed (How to resolve?)
testing port 9084... done
testing port 9086... done
testing port 9088... done
testing port 9090... full cone test failed (How to resolve?)
testing port 9092... done
testing port 9094... done
testing port 9096... done
testing port 9098... done
testing port 9100... done
testing port 9102... full cone test failed (How to resolve?)
testing port 9104... done
testing port 9106... done
testing port 9108... done
testing port 9110... done
testing port 9112... done
testing port 9114... done
testing port 9116... full cone test failed (How to resolve?)
testing port 9118... full cone test failed (How to resolve?)
testing port 9120... done
testing port 9122... done
testing port 9124... done
testing port 9126... done
testing port 9128... done
testing port 9130... done
testing port 9132... full cone test failed (How to resolve?)
testing port 9134... done
testing port 9136... done
testing port 9138... done
testing port 9140... done
testing port 9142... done
testing port 9144... full cone test failed (How to resolve?)
testing port 9146... done
testing port 9148... done
testing port 9150... full cone test failed (How to resolve?)
testing port 9152... done
testing port 9154... done
testing port 9156... done
testing port 9158... done
testing port 9160... done
testing port 9162... full cone test failed (How to resolve?)
testing port 9164... done
testing port 9166... done
testing port 9168... done
testing port 9170... done
testing port 9172... done
testing port 9174... done
testing port 9176... done
testing port 9178... done
testing port 9180... done
testing port 9182... done
testing port 9184... full cone test failed (How to resolve?)
testing port 9186... done
testing port 9188... done
testing port 9190... done
testing port 9192... done
testing port 9194... full cone test failed (How to resolve?)
testing port 9196... done
testing port 9198... done
testing port 9200... done
testing port 9202... done
testing port 9204... done
testing port 9206... full cone test failed (How to resolve?)
testing port 9208... done
testing port 9210... done
testing port 9212... full cone test failed (How to resolve?)
testing port 9214... full cone test failed (How to resolve?)
testing port 9216... done
testing port 9218... done
testing port 9220... full cone test failed (How to resolve?)
testing port 9222... done
testing port 9224... full cone test failed (How to resolve?)
testing port 9226... done
testing port 9228... done
testing port 9230... done
testing port 9232... done
testing port 9234... done
testing port 9236... done
testing port 9238... done
testing port 9240... done
testing port 9242... done
testing port 9244... done
testing port 9246... done
testing port 9248... done
testing port 9250... full cone test failed (How to resolve?)
testing port 9252... full cone test failed (How to resolve?)
testing port 9254... done
testing port 9256... full cone test failed (How to resolve?)
testing port 9258... done
testing port 9260... full cone test failed (How to resolve?)
testing port 9262... done
testing port 9264... done
testing port 9266... full cone test failed (How to resolve?)
testing port 9268... done
testing port 9270... done
testing port 9272... done
testing port 9274... done
testing port 9276... done
testing port 9278... done
testing port 9280... full cone test failed (How to resolve?)
testing port 9282... done
testing port 9284... full cone test failed (How to resolve?)
testing port 9286... done
testing port 9288... done
testing port 9290... full cone test failed (How to resolve?)
testing port 9292... done
testing port 9294... done
testing port 9296... full cone test failed (How to resolve?)
testing port 9298... done
testing port 9300... done
testing port 9302... done
testing port 9304... done
testing port 9306... done
testing port 9308... done
testing port 9310... done
testing port 9312... done
testing port 9314... full cone test failed (How to resolve?)
testing port 9316... done
testing port 9318... done
testing port 9320... done
testing port 9322... done
testing port 9324... done
testing port 9326... done
testing port 9328... done
testing port 9330... done
testing port 9332... full cone test failed (How to resolve?)
testing port 9334... done
testing port 9336... full cone test failed (How to resolve?)
testing port 9338... done
testing port 9340... done
testing port 9342... done
testing port 9344... done
testing port 9346... done
testing port 9348... done
testing port 9350... full cone test failed (How to resolve?)
testing port 9352... full cone test failed (How to resolve?)
testing port 9354... done
testing port 9356... done
testing port 9358... done
testing port 9360... full cone test failed (How to resolve?)
testing port 9362... done
testing port 9364... done
testing port 9366... done
testing port 9368... done
testing port 9370... done
testing port 9372... full cone test failed (How to resolve?)
testing port 9374... done
testing port 9376... full cone test failed (How to resolve?)
testing port 9378... done
testing port 9380... done

 

[ycit.ash]

Any thoughts at all?

 

[Alphabetic]

I would run through https://www.3cx.com/docs/pfsense-firewall/ again.

First thing you need to do is to get your firewall to pass before you move on.

 

[JohnS_3CX]

Judging by the reply, it looks like those ports fail because there is no reply (i.e blocked altogether or maybe lost packets)

If you have bad overall voice quality in addition the the above, you may be facing packet loss.

But you really need to define what you mean by bad quality. Some descriptors that may come along with packet loss are:

• Robotic/garbled voice
• Cutting out
• Static-type noise
• Silent periods

Note: I am not referring to bad quality as in low sampling rate or non-HD voice, but rather bad quality as in broken audio

 

[ycit.ash]

Bad quality in this case is that the audio is quite choppy at times. Cutting out at a regular interval during the call.

 

[JohnS_3CX]

I imagine that if you run the firewall checker again, you might get more failing ports, and not the same ones always. I'm not sure if your connection is the actual problem or the firewall itself but given the choppy audio (lost audio data) and the no replies on some ports it may indicate that the connection is bad/overloaded or the firewall is misconfigured or overloaded.

You could try eliminating the firewall altogether if this is possible (for testing) and see if you still get issues in which case you might be able to isolate it to the internet connection or physical connection.

 

[DSXVOICE]

Is the host machine virtual? If so, validate that the clock is working properly. Your issue is typical of virtual hosted PBX where the clock is not in sync.

 

[ycit.ash]

Yes the 3CX instance is a VM. When I logged into the OS it has the correct time down to the second. It is all set to my current timezone as well.

I have checked for packet loss but can find no evidence of that anywhere.

I have AT&T Business Fiber, and their modem/router is in full bridge mode with ALL functions turned off. The pfSense box gets a direct connection outside...

 

[JohnS_3CX]

Please go ahead and give some detailed information about your setup as per here:
https://www.3cx.com/community/threads/information-to-provide-when-requesting-help.67558/

 

[ycit.ash]

I did connect the VM straight to the WAN subnet, interestingly enough it still gave the error even after assigning the OS the external IP directly. Also, a new development here, I suddenly fail the 5060 port check. I haven't changed any NAT settings at all, it passes when I had it directly connected to WAN though...so I'm very lost here as to what is up. I also apparently had a client try to call me yesterday evening and the logs say it rang my phone but it never did, also says they left a VM but my VM is emtpy...

• 3CX Version - Enterprise Annual 16.0.4.493
• Server OS - Debian 9
• Is the 3CX Server Hosted and where? - NO
• Provisioning Method: STUN
• Trunk Provider - SKYETEL
• Has the Firewall Checker passed - NO
• Are custom Phone Templates being used: NO

 

[ycit.ash]

I've got a complete packet trace of the firewall testing process...I'm not sure what to look for and conversations of a failed port to a passed port look identical...

 

[JohnS_3CX]

Ok if you have a complete trace, this here will help you understand how the checker works and what to look for (ie. how it knows there is a port blocked or remapped):
https://www.3cx.com/docs/firewall-checker/

Some additional info: https://www.3cx.com/docs/troubleshooting-firewall-checker/


This however only serves to confirm what the checker found. The issue here is more on how your network is setup and how the traffic traverses your NAT.

What virtualization environment are you using at the moment and what are the specs of the host and the VM?

 

[bwitzig]

I have searched something exactly what I am experiencing but have not found it.

I have all required ports open and all ports are statically mapped when running firewall check most ports pass with no error. The issue is that a lot of random ports in the 9000-10999 range fail. Most seem to pass with the normal 'done' in green, but a bunch fail with the 'full cone test failure' error. All of the previous tests pass.

I have tried packet sniffing to find the issue, but the output is incomplete not showing all the ports in the range. I am using pfSense 2.4.4p3. The pfSense was a VM, but I have switched it to a physical box and I seem to be having slightly better performance on the physical box, but the 3CX FW Test still fails random ports in the 9000-10999 range.

I am having terrible call quality to boot...

I would recommend starting from the easiest to the hardest parts to troubleshoot.

How are the client's connected? 3cx app? Hard phones? Webapp?

If you enable pbx delivers audio and try to record a call do you hear the issue in the recording?

In our environment, we had to change some codec orders for calls to prevent transcoding which can greatly impact call quality.

 

[JohnS_3CX]

I would recommend starting from the easiest to the hardest parts to troubleshoot.

How are the client's connected? 3cx app? Hard phones? Webapp?

If you enable pbx delivers audio and try to record a call do you hear the issue in the recording?

In our environment, we had to change some codec orders for calls to prevent transcoding which can greatly impact call quality.

It was mentioned above that the audio is quite choppy at times - cutting out at a regular interval during the call.

It sounds more like a packet loss or possibly one of the elements in the chain were overloaded and hitting 100%