Port being changed before reaching PBX

[Peter Richardson]

Hi all, here's a tricky one for you!

I have been troubleshooting an issue with tech support for a while and it has reached the limits of my knowledge. Tech support have been very helpful, but I'm a little out of my depth. I get the concept, I lack the experience required to fix the issue.

The phones are connecting on a random port instead of their assigned SIP and RTP ports. This works for a while, but eventually the phones drop their connection and are unable to reconnect to the PBX (they just keep trying). I'm guessing that at this point they are being blocked by the firewall.

I have opened ports on the firewall on the PBX side (remote)
I have NATed the ports to the phones on the firewall where the phones are located
I am using pfSense firewalls at both ends
I have created the port preservation rules at both ends as per this link:
https://www.3cx.com/docs/pfsense-firewall/#h.mk14nzhjw26j

I know that 3CX doesn't support firewalls or any network configuration which is totally understandable because of the many different firewalls on the market, and their countless configuration options, and 3CX's (then implied) liability for your network. I'm just wondering if anyone out there might be able to spot something that I have overlooked or a slightly incorrect rule.

Let's just talk about one extension, the Yealink DECT cordless phone, this seems to be the one that causes the problem most of the time (it's rarely every any of the other phones that go down, but this is the most important phone to this client).

The extension is 103 and you can see from the 3CX Activity Log screenshot that it is connecting on port 25342 (am I reading that right?)

The Yealink interface screenshot shows that it is connecting on port 5060 for SIP, but shouldn't that be 5069 because that's what I have it set to in the PBX? (see attached)

I have attached images of pfSense rules and NAT for local site and remote PBX site.

I've run out of ideas, can anyone please suggest a place to start?

 

[Saqqara]

As above, why not create a site to site vpn using the pfsense routers. Make sure traffic can flow between the two networks

This way you configure the remote phones as local lan devices, and you should not have any issues.

 

[lneblett]

OK, one more time.

There are two (2) sides to every call. One side is at the PBX which is 5060UDP for SIP and 9000 to 9XXXUDP for RTP. 3CX is the SIP server and is using the range of RTP ports to accommodate the SIP Trunk provider and any remote extensions and because it is one device handling multiple calls, it needs the range of ports so that each call is on its own set of ports. These are the LOCAL ports for 3CX and this is why you port forward to same.

On the Phone Side, the phones are behind a router and there are multiple phones. The router may not handle the NAT/PAT correctly without some help. So, each phone has its own unique SIP port and its own unique set of RTP ports as assigned by 3CX (each phone is assumed to also be able to handle multiple calls as well). These are this side's (phones) LOCAL ports and are the ports used by each phone to listen for 3CX messages.

All the remote phones will send to 3CX using port 5060 for the call set-up and breakdown (SIP) and each will then send the audio using the ports in the 9000-9XXX range as directed by 3CX in the SDP headers contained within the SIP messaging. 3CX manages tells the phones what ports use.

Conversely, 3CX will send its SIP messaging to the remote phones using ports 506X to 50XX and on 14XXX-14XXX for RTP depending on the remote phone that is involved.

The LOCAL ports are what each side has open and forwarded. When the remote phones initially register with 3CX, each will tell 3CX something like please register me for an hour and should any call come to me, please use port 506X to let me know of it. When a call does come, 3CX will indeed use 506X and then tell the phone to send the audio on port 9XXX. The phone will respond to that INVITE with additional messaging which will include telling 3CX to send its audio back to the phone using port 14XXX.

As each side has an expectation of where the messaging will be received, they are listening only on these ports and this is why each router is forwarding these ports to insure correct delivery.

 

[Peter Richardson]

Hey @sip.bg and @Saqqara yes this sounds like an excellent idea. But I am trying to keep my setups as self-contained and simple as I possibly can. I am also trying to demystify some of the theories and gain some solid understanding so that I can create a setup that is rock-solid and troubleshoot issues when they arise. And lastly, I want to scale up to about 100 clients if I can, so I need to ensure each setup is reasonably similar and simple and easy to replicate. Does anyone here have this kind of experience with 100 or so clients? What do you use to keep track of everything? (documentation, billing, etc)

@lneblett thank you very much for your excellent and clear explanations, these have been extremely useful for me, you speak my language!

Questions:

1. How can I check to make sure that the phones are using the correct RTP ports?
2. Should I still use NAT Manual?
3. When using pfSense and 3CX together (at the PBX side), according to this guide: https://www.3cx.com/docs/pfsense-firewall/#h.mk14nzhjw26j it's important to set up Port Preservation in pfSense, so that pfSense doesn't change the port before it reaches the PBX (from what I understand). Is this also required on the phone side?
4. If I wanted to be proactive and check for any other issues or errors that might be happening that are not obvious or causing immediate issues (but possibly will in the future), what errors should I look for and where would I find them? (I'm just wanting to keep this client because they've had so many problems since we put this system in, I'd like to be proactive in fixing any other issues that are not immediately obvious to a novice like me!)

 

[sip.bg]

I have many customers using Layer2 (VLAN or MikroTik's EoIP) or Layer 3 (VPN) tunnels to their remote sites (or one customer to multiple sites). All tunnels are IPsec. This means phones are provisioned as local LAN (no NAT). Some users have remote phones using STUN, but these are either single ones (e.g. home workers) or in rare cases -- several phones per site. I prefer installing MikroTik routers where possible (prices start from US $20 -- for hAP lite / RB941 model).
3CX soft clients are using 3CX tunnel. You may use 3CX SBCs at sites you don't have control over router or still I prefer installing a MikroTik router behind some other existing router and use tunnels (the MikroTik router is default gateway for IP phones in such setup, i.e. the network topology is equivalent to SBC). At data center I use MikroTik cloud-core router, capable to handle multiple IPsec tunnels.



@Peter: If possible, build VPN tunnels using your pfSense routers and provision phones as local ones. Obviously pfSense are not doing NAT correctly, as you need different SIP port for each IP phone (this is limitation of the router, similar to situation where only one PC could browse internet, as this is happening at port 80 and PAT translation is public address:80 <=> internal address:80).

 

[Saqqara]

You say you want to upscale to 100 clients, but you have not mentioned how many phones at each location - this is the most critical info.

If you have multiply phones two or more phones , site to site vpn or 3cx sbc is the route to go down Instead of stun setup.

 

[lneblett]

As stated by many of us in different ways, each site has to be examined for its needs and then a course of action implemented that allows for those needs to be met. There are oftentimes different ways to accomplish the goal, but some ways are better than others and of course, the amount of money that one has available may influence which solution is implemented.

You need to consider the size, location, connectivity, functionality needed and costs for each site. There is no one size fits all. In some cases, a SBC may be the answer, others a VPN and even others may be best suited to having its own PBX that is bridged to others that can then be structured to offer some form of redundancy.

3CX tries to take much of the complexity out and when used in its simplest form, it is a breeze. When the complexities start to creep in such as non-supported phones, trunk providers, remote sites, SBCs and the like, this is when it gets a little more tricky and you start to gain some experience by trial and error, using the forum and last, but not least, reading the manuals of the equipment in use.

Each of us has our own way of doing things, none of which is wrong if it works, just different.

 

[Peter Richardson]

Thanks @sip.bg this looks epic, and I might look into this in the future. We've already made a lot of progress using pfSense, so I wouldn't like to start again with Mikrotik, although I've heard that they are awesome. But I'm sure pfSense can do all that you have suggested. It might require a bit more learning for me, but thank you very much for this excellent suggestion, makes a lot of sense!

@Saqqara I'm hoping to build a successful and profitable business that offers small to medium businesses in my city services such as hosted VoIP, web server, email server and similar services. The VoIP is the main thing at the moment, so I'm hoping to expand as fast as I can, gaining clients that need 5 - 20 phones on average. I'll try the 3CX SBC again with the next client and hopefully it will be more reliable than we have experienced thus far. When it worked, it worked perfectly, but when there were issues, nobody could solve them and it made the client rather angry.

Thanks @lneblett I agree. I'm going to start the 3CX training, I think this will help ensure I have a well-rounded understanding.

Anyone got any clues to my last few lingering questions?

Questions:

1. How can I check to make sure that the phones are using the correct RTP ports?
2. Should I still use NAT Manual?
3. When using pfSense and 3CX together (at the PBX side), according to this guide: https://www.3cx.com/docs/pfsense-firewall/#h.mk14nzhjw26j it's important to set up Port Preservation in pfSense, so that pfSense doesn't change the port before it reaches the PBX (from what I understand). Is this also required on the phone side?
4. If I wanted to be proactive and check for any other issues or errors that might be happening that are not obvious or causing immediate issues (but possibly will in the future), what errors should I look for and where would I find them? (I'm just wanting to keep this client because they've had so many problems since we put this system in, I'd like to be proactive in fixing any other issues that are not immediately obvious to a novice like me!)

 

[sip.bg]

Anyone got any clues to my last few lingering questions?

Questions:

1. How can I check to make sure that the phones are using the correct RTP ports?
2. Should I still use NAT Manual?
3. When using pfSense and 3CX together (at the PBX side), according to this guide: https://www.3cx.com/docs/pfsense-firewall/#h.mk14nzhjw26j it's important to set up Port Preservation in pfSense, so that pfSense doesn't change the port before it reaches the PBX (from what I understand). Is this also required on the phone side?
4. If I wanted to be proactive and check for any other issues or errors that might be happening that are not obvious or causing immediate issues (but possibly will in the future), what errors should I look for and where would I find them? (I'm just wanting to keep this client because they've had so many problems since we put this system in, I'd like to be proactive in fixing any other issues that are not immediately obvious to a novice like me!)

Answers to your questions:
1. Use wireshark or I could recommend sngrep for Debian installations (https://www.3cx.com/community/threads/sngrep-linux-tool-for-realtime-analysis-of-voip-traffic.50332/ -- replace jessie with stretch on Debian 9).

2. Generally yes, it is universal, but result may depend on routers you use (different routers do NAT in different way, especially for SIP communication, that's why 3CX recommends switching off SIP ALG, while FreePBX recommends switching on SIP ALG). Please note NAT is modifying only address header of an IP packet, while SIP ALG is modifying SIP details inside packet and also in related packets (using SDP).

3. For PBX site -- yes, you should. For phones site, if you do it, each phone must use different local SIP port and different sets of RTP ports. Usually SIP ports are 5060 - 5xxx, while RTP ports are above 10000. Each call uses one SIP port and two RTP ports, but the phone may require more (for each account). Protocol used is UDP, unless you use encryption (TLS), where SIP is using TCP.

4. Running PBX firewall checker and client's firewall checker (https://www.3cx.com/docs/firewall-checker-client/) will generally guarantee proper operation of the PBX and phones. If you experience firewall / NAT issues usual effects are one-way audio, not terminating calls after completion or prematurely terminating (typically at 32-nd second). Check making inbound and outbound calls with length more than 90 seconds and terminate call by caller and by callee to see everything happens as expected. Also check blind and attended transfer, conference calls (3-way conference, initiated by phone's menu), proper caller-ID being sent out and DDI call being received (at least at 2 different phones).

Hope this helps.

 

[sip.bg]

I still recommend using VPN tunnels instead -- then no NAT or port forwarding will be involved and you can provision phones as part of LAN.

 

[leejor]

we were using an SBC but it was causing too many problems when the power went off, so we had to get rid of it because nobody could find a way to make it work after the power came back on (reliably).

If power issues are the only reason you aren't employing an SBC then you might consider a UPS, if power outages are a regular occurrence.

 

[Peter Richardson]

Thanks @sip.bg

I'll have a go at learning how to use Wireshark and see if I can see the RTP ports in use.

Thanks for the other tips also, I have made many notes!

@leejor this is a great idea, I think I'll make them part of my standard setup from now on. Although we really shouldn't have to, they should just figure out how to make it work after the power goes out without user intervention - having said this, I just read about the latest update to the SBC which has some mention of a fix for when the power goes out.

The client called me today wanting to have a "meeting" to discuss whether to continue service, but I assured him that I have found and fixed the issue, which I believe I have. But today I also noticed three more things:

1. Extensions going unregistered and back to registered
2. Extension list showing "2" phones instead of 1 on 3 out of the 4 extensions. There is only one extension that is also using the PC client, but the other 3 are not.
3. Something that seems like the phones are still using an odd port.

 

[leejor]

Something that seems like the phones are still using an odd port.

If the sets are provisioned to use a port in the 50XX range, it may be a router that is changing them. What make/model routers are you using. Someone may be familiar with them, and can suggest chances to the settings, that may offer a better "experience". Not all routers "play well" with SIP without modifications.

 

[Peter Richardson]

Hi @leejor I'm using pfSense. Yes this has been the theme of this whole thread, having to configure the router correctly. We've made big progress, but I need to know what the code in those above images mean, whether it's still not working properly or if it's meant to say that. I don't understand SIP code yet.

 

[Peter Richardson]

anyone?

 

[Peter Richardson]

Hi all,

Still having these issues:

1. Extensions going unregistered and back to registered
2. Extension list showing "2" phones instead of 1 on 3 out of the 4 extensions. There is only one extension that is also using the PC client, but the other 3 are not.
3. Something that seems like the phones are still using an odd port.

and now today having more issues. One-way audio being the main issue. I noticed this on the activity log - the phones are connecting like [email protected]:5069 where 0.11 is that extension's own IP address. What the hell is going on?!?!?!

 

[sip.bg]

Peter,

I've advised you what to do and answered your questions, if you have read my posts. There is nothing more to say (from me or anybody else). Of course there is a chance of making some mistake and this to be the reason for some of your issues.

The simplest thing to do, if you want to keep your pfSense devices (routers) is to build VPN tunnels and avoid NAT / port forwarding. Obviously pfSense is not handling SIP traffic correctly and you can't compensate this behavior with different port tricks and STUN.

We are maintaining over 100 3CX and FreePBX cloud based installations with different network topologies at customer sides, including banks and commercial companies with over 100 branches / 1000 phones.

Regards,
Orlin

 

[Brad Cann]

Seriously, build a site to site vpn use pfsense (Pretty easy to do) and stop mucking around trying to go out one network over an ISP and back into another network. At least with a VPN you can control quite a lot of variables (not all of them) and its nice and secure.

Also once you eventually understand why a VPN is better, learn QOS, otherwise your calls will go crap and people will wonder why you recommended IP phones and VOIP, and you will wear the consequences.

 

[Peter Richardson]

@sip.bg you have been awesome, thanks very much. I am going to cave in and go with your suggestion. Obviously if you can make it work for so many customers, it's obviously the best way to go. And obviously as you and @Brad Cann have pointed out, it's simply not working the way it should with pfSense, for whatever reason(s) - all of which we don't have the time or patience to figure out.

 

[Brad Cann]

quite simply without a site to site VPN, your allowing your voice calls to go out onto the internet "With the great unwashed masses", and allowing any kind of packet snooping, packet loss and jitter to come into affect. Whilst you can't 100% control jitter with a VPN, you can certainly reduce the number of hops your packets go and put *SOME* level of QOS on them, even on a consumer grade connection, even if that is only to your router interface level....

 

[lneblett]

I guess the first question to ask, is whether or not the calls are working or failing? In an earlier post you indicated that the ports were correct and now, apparently not so, but there is not any mention of the impact to actual call activity.

The other post mentioned Yealinks, but one capture is showing a 3CX softphone. Not knowing how provisioned and/or if the provisioning matches that as in the extension settings for the softphone, is an unknown.

Some of other captures also seem to be related to the same extension as that of the softphone, but not enough info to tell.

A couple of things to look out for when using the softphone - the computer going to sleep or the NIC card using power saving features may cause issues with registration.

Also, did you check out anything about the firewall:
https://doc.pfsense.org/index.php/VoIP_Configuration

You will note that pfsense wants source port remapping enabled. I am uncertain how you have yours configured at either end, but you can use the link to learn more about what settings may be best. As you have separate SIP and RTP ports, there should be no need for source port remapping and it would normally be disabled.

I, like the others, suggest that a VPN may be your best bet to avoid the headaches involved in overcoming NAT issues. You will then need to re-provision the phones as if local. Also, if the internet connection is a normally available connection from a local ISP, a VPN does not lessen the hops nor provide any added QoS. In this case the VPN traffic is established over the open internet and the route is dictated by the providers who handle the traffic between the two points. The routing may change at anytime as the dynamics change - congestion, maintenance, outages, etc. QoS may be applied at the router level, but the second it hits the tunnel WAN to WAN, it is again on the open Internet and the providers not only treat the traffic (currently) as peer, but the data within the tunnel is encrypted so they would have no way to read the QoS tags anyway. However, the tagging may be preserved through the tunnel, so the receiving router may still be able to act upon it. No matter the technicalities, you should still examine the benefit of its use. Just be advised that if QoS is to be employed, the routers will not typically react to the tagging until such time as the router senses that the WAN side is becoming congested. So, sometimes, it may be better to simply leave it off and only consider its use if and when issues start to arise.