Firewall checker, security, and the 3cx helpdesk

[mpruett]

I support 3cx for myself and a few clients. I've worked with a 3cx partner on a variety of issues who makes tickets with 3cx on behalf of us. I'm not going to give details on the current issue we are working on, but I wanted to voice my concern and maybe vent a little bit about an issue we have encountered on multiple occasions.

I would make the claim that the firewall checker in it's current form, is bad security practice. Especially when combined with your support departments near refusal to troubleshoot any issues related or not until the 3cx install passes the check. The tool pretty much requires all ports that 3cx could conceivably use to be open, regardless of if the services you are currently using depend on them. I understand the point of having a tool like this, inevitably your phone support will handle mostly very basic calls, created by customers with minimal knowledge, and handled by tier 1 support that also have minimal knowledge. I get it. And yes, sip is an awkward protocol with differing requirements based on the sip trunk vendor and configuration. But at the same time, if the firewall checker is recommending every possibly used port open, and your support department demands this before they even read the ticket, you are in effect advocating for the widest attack surface possible. Sure you can open things up to 3cx specifically, temporarily, to pass the check, but I'm willing to bet most of your clients when faced with this just open everything recommended up and call it good. On top of this the local nftables ruleset is basic, and does not adapt to actual requirements of the system. Another issue this perpetuates is that most of your support engineers don't really understand what ports are required when. If the tooling on this were improved, your own support staff would be able to get issues resolved more quickly and customers would trend toward more sane firewall rulesets.

 

[Evolute IT]

@StefanW @Nick Galea

 

[Pierre_3CX]

Hello,
The firewall checker is a very important self-troubleshoot tool to verify that the networking configuration upfront of the 3cx System will let external traffic reach it, in particular from VoIP providers. It does not only check for open tunnel/SIP/RTP ports but also for the port mapping and NAT configuration which must be a Full Cone NAT (this ensures multiple calls from different sources can reach you at the same time without port changes), and absence of SIP ALG (which would alter SIP traffic and can cause one-way audio/no audio issues).

Note the list of ports checked may look long and give an impression of large attack surface - this is due to the large RTP port range (which is designed to accomodate all licenses sizes in sim calls), but note that these ports are not in permanent listening state, in fact they will only open two-by-two per call basis and close right after a call is ended, and will rotate in round-robin fashion through the range.

Furthermore if you want to filter by ACL your SIP/RTP ports and not let it open to all IP addresses for security reasons you can do that but you must be careful in ensuring you allowed:
- all IP ranges of your VoIP providers (otherwise you may lose calls or have no audio issues)
- all IP ranges of your remote sites (if any are having STUN-provisionned phones)
- the video conferencing MCU servers
- the servers used by the firewall checker so that it keeps functional (that would be the STUN servers and SIP ALG Detector domain).

Finally at support this is not a show-stopper but depending the issues experienced it will indeed be a valid request to confirm this passes so that common fail points are excluded and troubleshooting can continue, as we've seen many cases caused by firewall/router misconfigurations. Feel free to PM me if you would like us to review a particular case as I could not find any ticket with your contacts.

In regards to nftables you are saying it does not accomodate requirements of the system but that would be the case only if you added some third party services to the machine isn't it? as it opens by default all ports needed by the 3cx system.

 

[mpruett]

You seemed to just reiterate the points I made. The firewall checker doesn't check what is needed by the system as configured, it checks what would hypothetically be needed by every possible service under every possible configuration. The local nftables ruleset also similarly does not permit what is needed by the current configuration, but rather what could be needed.

For example my personal 3cx system has a trunk to telnyx using SRTP/TLS, it performs outbound registration, I have no ports natted inbound whatsoever, it works fine. However the firewall checker would fail over unused and unnatted udp ports.

"Finally at support this is not a show-stopper but depending the issues experienced it will indeed be a valid request to confirm this passes so that common fail points are excluded"

That hasn't been my experience across multiple tickets, at least not without days of back and fourth and arm twisting. For example in the current ticket I'm working on with another client we are trying to convert an existing telnyx trunk to tls. SRTP works, TCP transport works, however TLS fails immediately. Investigating packet captures (both in your gui tool, locally on the 3cx server using tcpdump, and on our firewall) shows that when TLS is selected no outbound traffic to sip.telnyx.com occurs at all, on any port. The logs show an error complaining about "resolution" (though dns is confirmed to be working fine). I'm not at the office at the moment so I can't elaborate, but the point is that the packet capture does not include anything involving sip.telnyx.com.

In the first ticket I supplied the support package, the logs and packet capture while trying to refresh the trunk registration, as usually requested, as well as a clear explanation of the issue and the log messages of interest. Their first response was to ask for the specific date and time the issue occurred in the logs. Despite me being clear on this and the packet capture literally being like a few mins long at most. If they had thought about this for more than two seconds or read the ticket they would know that what they had was very specific. So I responded to this effect. Then a day later they came back with something like "we see your firewall check fails, please fix that and resend the capture". Like actually wtf? They are asking for this, when the packet capture they already have, generated on the 3cx server, behind our firewall, contains no related traffic. In fact, and I highlighted this in the ticket, we also have a freshly installed stock 3cx server for testing this on the same network behind the same firewall, and with the same trunk configuration it connects just fine. It's *obviously* not an issue with networking, no attempt is even made to register.

That ticket though isn't really the point of my post here. My point is that support behavior like I described above is incentivized by a firewall checking tool that lacks any real granularity. It's testing for things well beyond what is actually needed, and this is inevitably going to cause end users to in-effect, misconfigure their firewalls from a security perspective. You can always make the argument that ports that are not listening are not exploitable, and that misses the entire point of a firewall. A firewall acts as an external enforcement policy for predefined network behavior. Ports that are not used, should not be permitted in the firewall, especially ports that could become open under certain configurations, it helps you define and detect what expected and unexpected behavior is.

The firewall checking tool should look at the 3cx servers configuration, the trunks and providers used, and generate a specific list of ports that are required. Report that to the user broken down by service. This would be far more useful to your support reps as well, and also serve to educate them on what is required when, rather than having them essentially recommending to end users a laissez-faire firewall policy, to proceed with a support request.

 

[Evolute IT]

For example my personal 3cx system has a trunk to telnyx using SRTP/TLS, it performs outbound registration, I have no ports natted inbound whatsoever,

That's the kind of thing that will "work" until it doesn't, thus the firewall checker.

Some firewalls/providers are NAT friendly, others aren't. Just because it works fine for you, it doesn't mean it's a general thing. On the contrary, most 3CX users probably respect the base port forwardings and avoid most issues this way.

It may not be the most "secured" from your point of view, but after using 3CX that way for 3 years, we never got hacked. Following simple best practices is the best weapon against hacks.

 

[SweetAction]

@mpruett
Keep in mind that 3CX is not built for one specific use case, it's built for many. It makes sense that the tools provided (FW checker in this case) are also targeted to every use case 3CX supports.

Would it be great if the checker in some way looked at current config and then tested just that? Sure. Sounds like some dev work. Until then, I can't blame 3CX for making a tool that covers all their use cases regardless of the specific implementation it's running on.

Sounds like your real gripe is Support's ability to... support. "Tier 1 reads off a script, ignores my case notes, yada yada yada". It's the same story with every vendor in every vertical in every space - rare exceptions apply. Either you staff support with experienced engineers and lock down support to only answer from experienced callers who pay a fortune to keep these engineers employed OR you staff with tier one, open to general public, give them a script to follow, and lower ticket costs. You have tier 1 escalated when the script ends. 80% of the time... they can solve it. I think we know how most companies set it up. I can't help you with this other then to advise patience and working through the system till you get where you want to go.

For your issue - as crazy as it will sound - reinstall the PBX and restore backup. I had a wacky case many moons ago similar to this and that was the solution. Take ya 15 minutes to get back up and running.