Outgoing Call Issue and Firewall Rules

[Steve Medley]

I have one 3CX site that started having outgoing call issues a few months in, I checked with our SIP provider and all of the outbound calls they can't make are not even showing on their end as an attempted call. They pick up the phone and just can't make an outbound call.

After checking routing, switching, phones and even replacing their modem I found something suspicious in the logs.

Over a 10 hour period they were getting 100+ authentication failures from a dozen different public IPs that weren't theirs. All of them traced back to DigitalOcean.

Authentication failed for AuthFail Recv Req REGISTER from xxx.xxx.xxx.xxx

I figured that someone found out port 5060, etc. was open on the WAN and was trying to force a way to authenticate on their SIP and use it for calls.

I called the SIP provider back and they said although they can't say for sure, all of those failed auth attempts might be responsible for the outbound call issue. So I changed our WAN>LAN firewall rule on the 3CX ports to only be allowed from a white list.

This whitelist was basically our SIP providers IP range, the FQDN of the three 3CX STUN servers listed in Settings:Network:External IP Configuration as well as the IPs that those three STUNs resolved to with nslookup just because I've had some issues with FQDNs and firewall rules in the past.

However this made the phone system get much worse, now inbound calls would ring but as soon as someone in the office answered, the caller would get disconnected immediately. I tested and confirmed this by changing the rule from allowing the white list only and calling in (getting disconnected) then changing it to allow all and calling in successfully.

So now I'm up against a wall and not sure what to do next. I don't even know for sure the intermittent outbound issue is caused by the failed auth attempts, but if it is how do I block this as they keep changing the IP they are coming from? Is there something I'm missing on my whitelist? Some way to not have allow all on the WAN>LAN? Anything else I am missing?

System Details:
3CX Professional 16.0.525
Phones: Yealink SIP-T29G on Firmware 46.83.0.60
SIP Provider: XCast (I'm aware they are unsupported, but they worked for months on this site and are still working perfectly on 10 other sites I've setup, never seen this issue before).

 

[cobaltit]

With any weird network issue you really just need to get a PCAP of a working and non working call and figure out what the difference is.

 

[Steve Medley]

With any weird network issue you really just need to get a PCAP of a working and non working call and figure out what the difference is.

It's not specific outbound calls. It's like an all or nothing on outbound, they all work fine, then they all stop working a few times a day for sometimes over an hour.

 

[cobaltit]

Re-reading the thread, you mentioned replacing the modem. Do they only have a modem/router combo and not a dedicated modem and dedicated router? What modem?

 

[JohnS_3CX]

Hi Steve,

I would suggest to go back to your original network and firewall settings to when before you made any changes, and temporarily disregard the provider's comment.

Then set your logs to Verbose and check to see what happens during the failed calls. A manual capture or a tshark background capture can help you see what happens exactly at the time of the failed call. If the provider does not see the attempt, your first job should be to see what the PBX does because it is the instigator of the call.

Meanwhile, your PBX will blacklist any IPs that try to hack your system after reaching a certain threshold.
I also recommend Going to Settings > Security and enabling Automatic Global 3CX IP Blacklist.

 

[Steve Medley]

Re-reading the thread, you mentioned replacing the modem. Do they only have a modem/router combo and not a dedicated modem and dedicated router? What modem?

Replaced the AT&T Fiber modem, but it's in Bridge Mode (Or at least as close as AT&T allows) to a SonicWall TZ300

 

[Steve Medley]

I would suggest to go back to your original network and firewall settings to when before you made any changes, and temporarily disregard the provider's comment.

I actually ended up having to do this, the firewall whitelist made it much worse so we are back to how it originally was. I downloaded the support info and used the log explorer but didn't have verbose mode on, so I will try that now.

 

[cobaltit]

So if it's the ABF fiber there's definitely an issue with a low amount of concurrent sessions in the state table that can cause this issue. It used to be pretty low (like 2000 sessions) and I think it was upped in a recent firmware to something like 8000 but even that isn't hard to hit.

 

[JohnS_3CX]

I'm not aware of the restriction that @cobaltit mentions above for the specific provider, but if he has come across it then I trust this has happened to others before.

Either way, you should be able to isolate and prove this via a sonicwall packet capture, where the PBX will clearly be sending the request, the sonicwall will be allowing it to pass, and the provider will be able to verify from their logs that it never reached them. The only thing standing between their SIP provider and the sonicwall is hopefully just the ISP now.

 

[Steve Medley]

So I've gone through regular 3CX logs, support info logs, verbose logging PCAPs and my SonicWall PCAPs, I think I'm starting to identify a pattern of what seems to "kick in" this issue.

Users begin dialing a number, say 919-555-1212, but hit send by mistake part way through, so an attempt to dial 1-919-555 is sent through the system. This occurrence seems to come right before the outage every time. This 6 digit number of course fails, but as soon as it does every single phone (including 3CX mobile apps) can no longer make outbound calls, sometimes only for 10 mins, sometimes much longer.

In the most recent case, a 6 digit mistake number was sent at 1:13PM PST, then 2 attempts to dial the full number were made, all three of these show in 3CX as Not Answered. BUT the sonicwall PCAP only sees the first 6 digit attempt and doesn't see the 2nd two attempts to make the call, so it's never even being sent to the SIP provider.

Is this a bug? Is there something I can do with an outbound rule that may act as a sort of workaround?

 

[Steve Medley]

Additional Info:

We also confirmed it happens when you dial the in / out of office buttons which dial *641 & *642, so it appears to happen with anything that's not a full phone number.

 

[Steve Medley]

Is there anyone who's experienced anything like this? I can't find any other threads or articles about something like this.

 

[cobaltit]

Never come across anything like this. You mention the Sonicwall PCAPs don't show the attempts. Do you have 3CX PCAPs at the same time showing the attempts? And do you have an outbound rule sending those mis-dialed numbers out your trunk or are they being rejected by 3CX?

 

[JohnS_3CX]

Hi Steve,

This occurrence seems to come right before the outage every time. This 6 digit number of course fails, but as soon as it does every single phone (including 3CX mobile apps) can no longer make outbound calls, sometimes only for 10 mins, sometimes much longer

You should reject this as a mere correlation, it does not imply causation in any way. Here is what this would look like in your logs if a user dialled something the system does not have a matching rule for:

02/10/2019 11:19:32 - [Flow] Target endpoint for 1919555 can not be built!

Call was simply blocked for not matching any rules, all else remains unaffected.

We also confirmed it happens when you dial the in / out of office buttons which dial *641 & *642, so it appears to happen with anything that's not a full phone number.

Again, only correlation here - nothing to imply causation.

In the most recent case, a 6 digit mistake number was sent at 1:13PM PST, then 2 attempts to dial the full number were made, all three of these show in 3CX as Not Answered. BUT the sonicwall PCAP only sees the first 6 digit attempt and doesn't see the 2nd two attempts to make the call, so it's never even being sent to the SIP provider.

This is exactly why you would need to compare the firewall PCAP against the PBX one, and not just look at the plain old call logs. How do you know whether the PBX sent it out or not?

Note that Not-answered would appear if you dialled an invalid number, but it would also appear if you dialled a valid one and the other end did not pick up (ie. you hang up) hence it is the wrong tool for the job in this case. Focus more on comparing Firewall VS PBX pcaps