Solved One Way Audio on Queue Answer

[Blackstone Ops]

Hi,

I have a strange issue and I'm just looking for pointers on where to start looking.

If a caller dials an extension, everything is fine
If a caller dials a queue and the SAME user answers, then the INCOMING audio is dead i.e. The caller can hear the user, but the user cannot hear the caller.

The queue is on a separate DDI to the "main" number, if that's important. Both terminate at the same 3CX instance.

The user is using the 3CX soft phone, and is connected via a VPN connection.

We are running 16.0.9 on Debian. The issue started after we upgraded our Fortigate FortiOS from 6.4.3 to 7.0.1. No issues prior to the update. Fortinet support are completely stumped.

As a temporary solution I have disabled the ALG, opened the ports, etc etc. I am able to make and receive calls.

What I'm really trying to work out, is what is different between a caller dialing an extension, and a user answering a call from a queue. There is something in the difference that is causing our issue, and I suspect that difference is an unreported bug in the 7.0.1 firmware.

Many thanks for any pointers,

Geoff

 

[Blackstone Ops]

Hi again,

I know this isn’t a Fortinet support forum, and I’m not asking for that. But I did trawl the known issues with FortiOS 7.0+ and found this:


VoIP Bug ID: 682983
Description
SIP ALG does not DNAT all IP addresses in the SIP response messages (route field).

I know the usual advice is “disable sip-Alf” but as I have said, it used to work. So I’m leaning towards this as being the root cause of the issue.

 

[ChrisC_3CX]

To first gather a bit more information regarding this issue:

If a caller dials a queue and the SAME user answers, then the INCOMING audio is dead

Does this only happen if the same user answers or did you just mention it as proof that the extension itself is ok since direct calls to it work?

The user is using the 3CX soft phone, and is connected via a VPN connection.

Which 3CX soft phone is being used(3CX Windows App, 3CX mobile client, etc). Is the VPN you mention a VPN that connects the user to the same site as the 3CX PBX or is it to an unrelated network. Could you try disconnecting the VPN client completely and then access the 3CX Webclient of that extension to check if the issue persists? Make absolutely sure you have completely disconnected the VPN before launching the Webclient.

I know the usual advice is “disable sip-Alf” but as I have said, it used to work. So I’m leaning towards this as being the root cause of the issue.

Just to clarify, is SIP ALG currently disabled?

Also,
• Is this 3CX Instance running on a Debian 9 machine?
• Is this issue only affecting one specific user and one specific Call Queue?
• Is there any possibility you can test this with a 3CX Client that is on the same LAN as the PBX? Do you get the same behavior?

 

[Blackstone Ops]

Thanks for the response @ChrisC_3CX

If a caller dials a queue and the SAME user answers, then the INCOMING audio is dead

Does this only happen if the same user answers or did you just mention it as proof that the extension itself is ok since direct calls to it work?

Mentioned only as proof that the extension works. All extension-dial calls work. No queue-transfer calls work.

Which 3CX soft phone is being used(3CX Windows App, 3CX mobile client, etc). Is the VPN you mention a VPN that connects the user to the same site as the 3CX PBX or is it to an unrelated network. Could you try disconnecting the VPN client completely and then access the 3CX Webclient of that extension to check if the issue persists? Make absolutely sure you have completely disconnected the VPN before launching the Webclient.

All the clients I am testing with are using the 3CX Windows App. The VPN connects to the network that they require to access the PBX. I have also replicated the issue on site, without the VPN in the mix. The PBX lives in a separate firewalled zone on the Fortigate.

Just to clarify, is SIP ALG currently disabled

SIP-ALG is currently disabled so that users can work. With ALG off and the ports opened manually, everything works. The key point (for me) is that prior to upgrading the Fortigate, ALG was enabled, and everything was working just fine. What I'm hoping to do, is work out why a Queue answer fails with ALG on, when all other traffic seems to be OK.

• Is this 3CX Instance running on a Debian 9 machine?
• Is this issue only affecting one specific user and one specific Call Queue?
• Is there any possibility you can test this with a 3CX Client that is on the same LAN as the PBX? Do you get the same behavior?

Debian 9
All users, all queues.
The PBX is in a dedicated Zone on the fortigate. There are no issues inside that zone. Using the 3CX Windows App on the general network (which is connected to the same Fortigate, different zone) I have the same issue.

To recap: With ALG disabled and manual ports opened, there are no issues. If I enable the previously-working ALG, I only have an issue with calls answered from a queue. The only change was an upgrade from FortiOS 6.4.3 to 7.0.1. Fortinet are unable to explain why.

 

[ChrisC_3CX]

I must say, I can't be entirely sure of what changed with the firewall's firmware upgrade to cause the issue but everything points to a change in its SIP ALG feature. SIP ALG is known to cause such issues as it can modify SIP headers, and, for this case specifically, it may have something to do with the SDP body being modified since this is what carries the information regarding the media exchange. There are some differences in regards to what phase of the SIP Signaling the SDP gets delivered for a Queue call so that might explain why you are only facing this issue with those specific calls, however, whatever the case, it still does not justify it causing issues. For this reason, we always recommend that SIP ALG is disabled and this is what I recommend for this case as well.

 

[Blackstone Ops]

Thanks again @ChrisC_3CX

There are some differences in regards to what phase of the SIP Signaling the SDP gets delivered for a Queue call

Are you able to expand on this at all?

I understand the recommendation to disable the SIP ALG. If that's my only recourse, I will keep it disabled, but the increased complexity in the firewall rules, and decreased security with the additional open ports, is not something that I am keen to rush to.

 

[Blackstone Ops]

If I need to keep ALG off, I'd like to reduce the ports that I leave open. I followed this:

https://www.3cx.com/docs/ports/
and
https://www.3cx.com/docs/manual/firewall-router-configuration/

and opened 5060 TCP/UDP and 9000-10999 UDP, but it didn't work. Looking at the traffic logs I can see ports in the range of 42000 are in use.

I've seen other posts about opening UDP 42000-43000 and UDP 7000-8499, but I can't find any reference to these ports in the documentation. Are these specific to the 3CX windows app?

 

[Blackstone Ops]

@ChrisC_3CX I've been watching the logs on the Fortigate, and I noticed the following:

3CX Windows App -> 3CX PBX uses ports 443 and 5060
3CX->3CX Windows App uses port 42000-43000


I'm not seeing any SIP traffic (port 5060) from the PBX to the Windows app. Have I configured the 3CX windows app clients wrong somehow?

 

[ChrisC_3CX]

I understand the recommendation to disable the SIP ALG. If that's my only recourse, I will keep it disabled, but the increased complexity in the firewall rules, and decreased security with the additional open ports, is not something that I am keen to rush to.

I understand your concern, however, unless the firewall configuration is implemented as per our guides, I'm afraid we can't guarantee normal functionality. That means that all mentioned ports, must be configured on the 3CX PBX and services such as Port remapping, SIP, ALG, etc must be disabled.

Regarding the 3CX Clients, do bear in mind that they all perform an outbound connection to the 3CX PBX, in most cases, if not all, from a random source port and to ports specified in the 3CX required ports guide. That said, if these ports have been configured and there are no strict restrictions on the remote site where the 3CX clients reside, there should be no issues and no other port configuration should be required.

I'm not seeing any SIP traffic (port 5060) from the PBX to the Windows app. Have I configured the 3CX windows app clients wrong somehow?

This is probably due to the fact that the 3CX Windows App is remote to the 3CX PBX and therefore connects via 3CX's Tunnel connection which uses Port 5090 and encapsulates all SIP traffic. The 3CX Windows App will also need to contact the 3CX PBX on it's https port which by default is either TCP port 5001 or 443.

When I mentioned differenced regarding the SDP in a Queue call, I was referring to the fact that the SDP body is not included in the SIP INVITE from the Queue to the Agents, rather, it is included in the ACK SIP message later on.

 

[Blackstone Ops]

Thanks @ChrisC_3CX

The port documents do not mention the 42000-43000 range, but I am seeing traffic from the PBX to the Windows Apps in that range. No other traffic in that direction.

I do see SIP and HTTPS traffic from the Windows apps to the PBX.
I’m not seeing any traffic on 5090.

I’m starting to wonder if I’m running the Windows Apps in a weird way.

 

[ChrisC_3CX]

The port documents do not mention the 42000-43000 range, but I am seeing traffic from the PBX to the Windows Apps in that range. No other traffic in that direction.

Those are probably the 3CX Windows App source ports for a connection that got initiated from the 3CX Windows App, meaning that you should not need to configure those ports if everything I mentioned in my previous response applies.

I do see SIP and HTTPS traffic from the Windows apps to the PBX.
I’m not seeing any traffic on 5090.

Is the 3CX Windows App local or remote to the 3CX PBX?
If it's local, SIP Signaling Traffic from the 3CX Windows App will be sent to port 5060 of the PBX. If it's remote, the 3CX Windows App will connect to 3CX's 5090 port, establish a tunnel connection and then send SIP traffic through that.

I’m starting to wonder if I’m running the Windows Apps in a weird way.

If the client is physically remote to the PBX but you're seeing traffic on port 5060 it might be due to the VPN which makes local routing possible between the 3CX Windows App and the PBX.

 

[Blackstone Ops]

Thank you for the continued responses @ChrisC_3CX

Those are probably the 3CX Windows App source ports for a connection that got initiated from the 3CX Windows App, meaning that you should not need to configure those ports if everything I mentioned in my previous response applies.

Nope, these are most definitely initiated by the PBX. They are hitting egress rules that I had to set up, which means they're not in response to anything.
I shall go over everything in your previous replies to make sure that I have everything set up in the "new for me" way. (We've been running 3CX in the previous configuration for many years with no issues)

Is the 3CX Windows App local or remote to the 3CX PBX?

Logically local - that is to say, on a network segment that is reachable directly from a VPN connected PC. Topologically it is on a dedicated network (10.10.10.0/24) that is separate from the VPN network (10.1.0.0/24) but routing rules are pushed out the VPN clients, and firewall policies are in place, such that they can access the 3CX server using the ports in your previous responses. In addition, I had to add the 42000-43000 ports to the reverse policies, otherwise it doesn't work.

If it's local, SIP Signaling Traffic from the 3CX Windows App will be sent to port 5060 of the PBX

I see this traffic hitting the rule I have set up

VPN which makes local routing possible between the 3CX Windows App and the PBX

Yes, this is so. But without the addition of the 42000-43000 UDP ports in the PBX->VPN rule, we get one-way audio on when answering a call in a queue.

EDIT: I should note that we have 3CX Windows Apps on the LAN segment, which are exhibiting the same requirement for port 42000-43000 to be opened from the VoIP network to the LAN. So I know it's not a VPN-specific issue.

 

[Blackstone Ops]

Those are probably the 3CX Windows App source ports for a connection that got initiated from the 3CX Windows App, meaning that you should not need to configure those ports if everything I mentioned in my previous response applies.

Nope, these are most definitely initiated by the PBX. They are hitting egress rules that I had to set up, which means they're not in response to anything.

I should re-phrase this. The connections to UDP port 42000+ is being initiated from the PBX. That is, the connection is egress from the VoIP network, ingress to the network of the requesting Windows app (in the examples above, a VPN-connected 3CX windows app) and so requires firewall rules to permit the traffic.

 

[Blackstone Ops]

Hi @ChrisC_3CX

Well, I well and truly went down the rabbit hole on this one. I set up the rules per the documentation you linked and still had issues.

So I took a different path, and I have allowed `all` from the VPN to the VOIP zone. I removed all the reverse rules (so my security concern is dealt with). And guess what - it all works perfectly.

So there is some port (or ports) or service that are missing from the documentation, OR the PBX is reacting to our topology in a way that makes it less predictable, when the network is locked down.

What is strange, is that the only traffic I see passing through the "all" rule, is HTTPS and SIP - both of which were allowed in the locked down rule.

I'm comfortable with leaving the "all" in place, as it's from the secure to the insecure. I'll watch the traffic, and see if I can lock it down later in the week. I've created a rule to allow HTTPS and SIP, so anything that hits the ALL rule should be easy to identify.

Thanks for all your help, and bearing with me. My understanding of ALG was that it was facilitating the pinholes we required for the reverse traffic. That was clearly wrong on my part, because it works just fine.

 

[Blackstone Ops]

EDIT: Original message removed. My "fully solved" was erroneous, as my testing station had a special "all" firewall policy.

I have a policy that allows HTTPS, SIP, PING and UDP 9000-10999 to the PBX, but it doesn't work unless I add a rule that allows "ALL"

 

[Blackstone Ops]

@ChrisC_3CX I finally have some additional traffic hitting the ALL rule. It looks like there are requests from the user to the PBX to ports in the 7000 range. Those aren't mentioned in the ports documentation you linked.

EDIT:

I looked at the PBX config, and I can see:

FIRSTLOCALPORT: 7000
LASTLOCALPORT: 8499

I'm going to allow those ports and try again.

EDIT 2:

That did it. I replaced the rule for UDP 9000-10999 with UDP 7000-8499 and I am now able to remove the ALL rule with no issues.

Is this normal/expected behavior? I'm not seeing those "LOCALPORT" ranges anywhere in the documentation.

The VPN->VoIP rule I ended up with is ALLOW:

• PING
• SIP
• HTTPS
• UDP 7000-8499

 

[NickD_3CX]

@ChrisC_3CX I finally have some additional traffic hitting the ALL rule. It looks like there are requests from the user to the PBX to ports in the 7000 range. Those aren't mentioned in the ports documentation you linked.

EDIT:

I looked at the PBX config, and I can see:

FIRSTLOCALPORT: 7000
LASTLOCALPORT: 8499

I'm going to allow those ports and try again.

EDIT 2:

That did it. I replaced the rule for UDP 9000-10999 with UDP 7000-8499 and I am now able to remove the ALL rule with no issues.

Is this normal/expected behavior? I'm not seeing those "LOCALPORT" ranges anywhere in the documentation.

The VPN->VoIP rule I ended up with is ALLOW:

• PING
• SIP
• HTTPS
• UDP 7000-8499

Great to hear that!

Indeed, for the communication with other "Local" endpoints (on a LAN IP, not over internet), the 3CX Server will use the UDP 7000-8999 port range for RTP/Audio traffic.
I would still recommend though amending to your ports 10500-10999. Sometimes, if you have users that use the WebClient, it might use this port range.
If you don't do it, just keep in mind in case you run into this.

Indeed, the documentation does not say it as these ports don't need to be opened on the Firewall, but in your case, they did not to be included in the rule for your VPN to work.