STUN resolution not working

[Basti M.]

Dear community,

i have a strange issue with my STUN resolution. My logs show the following:

03/16/2020 8:40:10 PM - [CM506001]: STUN request to resolve SIP external IP:port mapping is sent to STUN server 51.38.45.26:3478/UNKNOWN_TRANSPORT fk=0 tgt= over Transport 0.0.0.0:5060/UDP fk=0 tgt=
03/16/2020 8:40:10 PM - STUN server stun2.3cx.com has IP: 54.39.182.217:3478/UNKNOWN_TRANSPORT fk=0 tgt=
03/16/2020 8:40:10 PM - STUN server stun.sipgate.net has IP: 217.10.68.145:3478/UNKNOWN_TRANSPORT fk=0 tgt=
03/16/2020 8:40:10 PM - STUN server stun.sipgate.net has IP: 217.10.68.152:3478/UNKNOWN_TRANSPORT fk=0 tgt=
03/16/2020 8:40:10 PM - STUN server stun-eu.3cx.com has IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=
03/16/2020 8:40:10 PM - STUN server stun-eu.3cx.com has IP: 51.38.45.26:3478/UNKNOWN_TRANSPORT fk=0 tgt=
03/16/2020 8:39:09 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/16/2020 8:09:09 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/16/2020 7:39:08 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/16/2020 7:09:07 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/16/2020 6:39:07 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.

Correct dynamic public IP gets displayed on dashboard and in parameter "DEFAULT_CONFIGURED_PUBLIC_IP", but FQDN won't get updated.
If I reboot the virtual machine, or restart the services, my 3CX FQDN is updated, but will not upon next ip change.

I created a packet dump with tcpdump for the STUN queries and everything looks normal. Even verbose logging won't show any leads.
Only thing that I don't understand is the "UNKNOWN_TRANSPORT" in those log entries.


Anyone an idea?

Thanks Basti

 

[JohnS_3CX]

Hi,

Start here please: https://www.3cx.com/community/threads/information-to-provide-when-requesting-help.67558/

 

[Basti M.]

System is hosted behind a Sophos UTM with ruleset corresponds to 3CX docs.
Tried with the default given STUN servers (stun-eu.3cx.com, stun2.3cx.com, stun3.3cx.com) and with multiple other STUN servers (e.g. stun.sipgate.net)

• 3CX Version: Standard 16.0.4.504
• Server OS: Debian 9
• Is the 3CX Server Hosted and where? Locally hosted on a Hyper-V Server 2019
• IP Phone Make/Model/Firmware: Should be irrelevant, but only 3CX Client for Windows
• Provisioning Method: Local
• Trunk Provider: wilhelm.tel (but should also be irrelevant)
• Has the Firewall Checker passed: YES
• Are custom Phone Templates being used: NO

Anything else needed?

 

[JohnS_3CX]

Is your PBX behind a dynamic IP?
Are you having any issues with external calls or remote extension?s

 

[Basti M.]

Yes it is behind a dynamic IP.

Remote extensions are not working. Trunk is currently not in use, but might also not work.
(It is working when i restart all services and the FQDN gets updates once, but after dynamic IP change the system does not change the FQDN anymore)

 

[JohnS_3CX]

Hi Basti,

Please note that the FQDN takes up to 6 hours to update whenever the IP changes due to the Standard License. https://www.3cx.com/docs/fqdn-management-allocation/#h.a31p7g6iztpp

Manual restart forces the IP to update the FQDN, but under normal usage there will be a delay of 6 hours on Standard License.

You have some options here:

1. If your IP changes fairly often, you should either consider getting a static IP which is the best solution for you and probably the most cost effective one.

2. Use a Pro Licence and manage your own custom FQDN, you can control you TTL and can update the IP as often as you want.

3. Use an Enterprise license which updates the DNS entry within a few minutes (no custom FQDN required)

Note 1: Unknown Transport is not something to worry about, the resolution still works
Note 2: Changing the STUN servers will not make any difference here, keep the default ones and the message "no valid STUN server" should go away.

 

[Basti M.]

Hi John,

1. I did knew that. My IP is changing once a day at 4 am. But I have no FQDN update within the whole day.

2. Not an option (private installation)

3. Not an option (private installation)

Even with the default 3CX STUN servers no update happens. So any different STUN server than 3CX is invalid?

 

[Basti M.]

With STUN servers set all to the 3CX ones as suggested above the log looks like this:

03/18/2020 11:26:42 AM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 11:06:42 AM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
03/18/2020 10:46:42 AM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 10:26:42 AM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
03/18/2020 10:06:42 AM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 9:46:42 AM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
03/18/2020 9:26:42 AM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 9:06:42 AM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
03/18/2020 8:46:42 AM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 8:26:42 AM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
03/18/2020 8:06:42 AM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 7:46:42 AM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
03/18/2020 7:26:42 AM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 7:06:42 AM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
03/18/2020 6:46:42 AM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 6:26:42 AM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored

Nothing has changed in behavior, still no FQDN update, even after 8 hours.

 

[JohnS_3CX]

"So any different STUN server than 3CX is invalid? "
Not necessarily, but we can at least guarantee that our ones will work with our software because we test them.

One thing I'm not very confident about is whether the error "no valid STUN server specified" was caused when you were troubleshooting and had changed your STUN server or maybe changed something else (for example if you modified any parameters that the system did not expect).

If you have no FQDN update for a whole day when using our STUN servers specifically, then this would indicate that something else is the issue. You might need to generate a Support Info file and open a ticket with 3CX Support in this case for further investigation.

 

[Cjay]

@JohnS_3CX
I don't want to hijack this thread, but the STUN issues are remarkably like what I reported in this thread not so long ago. I'm also on a dynamic IP and reliant on STUN & my activity logs are full (output every 30 mins) of these entries:

18/03/2020 13:11:27 - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.

This is clearly wrong since the system has resolved my external IP (shows correctly on dashboard) and 3CX provided FQDN resolves as expected. Everything works fine except for the logs being full of this!

 

[Basti M.]

Oh maybe it is because they have changed the available 3CX FQDNs for germany from "myfqdn.3cx.de" to "myfqdn.my3cx.de"??

But i do not see the support in this case at all. Why should I spent money on a bug?

 

[JohnS_3CX]

To change the FQDN you need to release the FQDN from the customer portal, and then reinstall the PBX using a backup that does not include license key and FQDN. At that point you will be asked to select a new FQDN.

If it was not done this way there may be issues. Can you confirm please?

 

[Basti M.]

I've now reinstalled the PBX and restored my cfg as described. But this could not be the solution. What are you doing with enterprise customers in standard licensing mode? If this is your solution, 3CX is not ready for production.

 

[Basti M.]

After reinstall:

03/18/2020 8:34:31 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 8:14:31 PM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
03/18/2020 7:54:31 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 7:34:31 PM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
03/18/2020 7:14:31 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
03/18/2020 6:54:31 PM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
03/18/2020 6:34:31 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.

Will update tomorrow if FQDN update works.

 

[Basti M.]

So, it was working a couple of days since reinstall, but stopped working again.
Now i've done nothing to the system at all since reinstall. No config changes, no changes in infrastructure, nothing.

Here are my logs:

04/13/2020 1:35:16 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
04/13/2020 1:15:28 PM - [CM506004]: STUN request to STUN server 51.38.45.26:3478 has timed out; used Transport: 0.0.0.0:5060/UDP fk=0 tgt=
04/13/2020 1:15:25 PM - [CM506004]: STUN request to STUN server 51.38.45.26:3478 has timed out; used Transport: 0.0.0.0:5060/UDP fk=0 tgt=
04/13/2020 1:15:22 PM - [CM506004]: STUN request to STUN server 51.38.45.26:3478 has timed out; used Transport: 0.0.0.0:5060/UDP fk=0 tgt=
04/13/2020 1:15:19 PM - [CM506004]: STUN request to STUN server 51.38.45.26:3478 has timed out; used Transport: 0.0.0.0:5060/UDP fk=0 tgt=
04/13/2020 1:15:16 PM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored
04/13/2020 12:55:16 PM - [CM306002]: There is no valid STUN server specified! External IP can not be resolved.
04/13/2020 12:35:28 PM - [CM506004]: STUN request to STUN server 51.38.45.26:3478 has timed out; used Transport: 0.0.0.0:5060/UDP fk=0 tgt=
04/13/2020 12:35:25 PM - [CM506004]: STUN request to STUN server 51.38.45.26:3478 has timed out; used Transport: 0.0.0.0:5060/UDP fk=0 tgt=
04/13/2020 12:35:22 PM - [CM506004]: STUN request to STUN server 51.38.45.26:3478 has timed out; used Transport: 0.0.0.0:5060/UDP fk=0 tgt=
04/13/2020 12:35:19 PM - [CM506004]: STUN request to STUN server 51.38.45.26:3478 has timed out; used Transport: 0.0.0.0:5060/UDP fk=0 tgt=
04/13/2020 12:35:16 PM - There's another STUN server that resolves to the same IP: 54.37.20.144:3478/UNKNOWN_TRANSPORT fk=0 tgt=; ignored

Btw i can see from tcpdumps, that 3CX is getting valid answers from the STUN requests. There is no timeout.

The dashboard is now also showing a wrong old ip address.

As this is so frustrating, i've now wrote my own application, requesting the same STUN servers as configured inside 3CX and updating the record when a change got detected. It is running as a daemon on the 3CX VM with no problems at all. So my guess is, that you have a bug somewhere in your code.

 

[JohnS_3CX]

Do you see from tcpdumps, that 3CX is getting valid answers from the STUN requests, or do you see that the NIC is getting them?

What the NIC receives and what the PBX receives can be two different things, as the OS can drop packets depending what is in the IP tables (or if you have any other software running on the machine).

In addition, there are some considerations regarding Hyper-V installations (with regards to NIC too). Can you confirm if this guide was followed?
https://www.3cx.com/docs/installing-microsoft-hyper-v/

 

[Basti M.]

Yes that might be something to check about. Is there any way of doing a packetdump from inside 3CX itself?

But until running my own software for a solution, the VM was stock installed from the latest 3CX ISO, so nothing changed in the system. If there is an issue with the iptables it will be with the stock ruleset. And shouldn't i see blocked packets somewhere in the linux journal?

Hyper-V is setup as required, static mac, VM gen 1 with its default adapter type (no legacy), VMQ is disabled on the network card even though it is an intel chip on the hypervisor.

 

[JohnS_3CX]

Hi Basti,

You can run a packet capture from the Activity Log page, but this is on the NIC level, not on the application level I'm afraid. Also if you are running our latest ISO file you should be ok on that end too regarding the IP tables (download latest and compare if you want to be sure).

If you want you can run a capture from your Debian terminal, and then restart all the services in the management console. Let the capture run for at least 5 minutes after the services all come up. Send me a PM with a link to download your capture and we can take a look from here too.

 

[Basti M.]

Hi John,

as seen above, restarting the services fixes the error temporarily. So a packet capture from the restart might not be that useful.

And the "There's another STUN server that resolves to the same IP" point to some working STUN resolution as you have already pointed out.

Something else about the STUN process. It seems that you are using some kind of ugly dns lookup for the STUN servers. In my packet dump the 3CX services always use a sequence dns transaction id (first server: 0x0001, second: 0x0002, third: 0x0003), this might lead into security issues and also might trigger intrusion detection systems. The transaction id should always be random.