Automatic Firmware upgrade

[mOrbo]

Hi,

2 days ago a new Snom (D385) Firmware was released. With earlier Firmwarereleases we had to deploy the Firmware to at least one device manually (Phones -> Firmware -> OK). After this, all other devices did the Upgrade automatically. This is not a very good method, because you can't test a new Firmware on some special devices in the IT-Department before deploying it to all users.

This time, even worst, the upgrade started automatically WITHOUT deploying the Firmware to one dives. This happened on three different installations. The Firmware was deployed to all Phones over night. Why is that? This should definitely NOT happen! All we did was SEARCHING for updates under "Settings -> Updates". We did definitely not deploy the Firmware under "Phones -> Firmware".

There MUST be a professional way to deploy Firmwares to single Phones to test the functionality properly and deployment shoud ONLY work manually (or with a check "automatic upgrade" like for the Systemupgrades). I remember a faulty Firmware from Snom some time ago with a broken phonebook after the Upgrade. Without testing possibilities and automatic upgrades its just a mess for bigger installations.

 

[JohnS_3CX]

Hi @mOrbo,

The way this works depends however on the way the manufacturer designed it, hence why some brands of phones behave this way (ie. Snom) whereas others require explicit deployment per phone (ie. Yealink).

If you wish to sample a firmware when it comes to Snom, you can download it off our website, upgrade a phone manually, and remove the provisioning URL temporarily so the phone will stay on the firmware you wish to test.

It is a bit strange however that the phones updated the firmware automatically, even with Snom you definitely need to deploy this at least once. I have tested one of our systems here and the firmware did not automatically deploy.

Did you perhaps have this option enabled by any chance?

 

[mOrbo]

Hi John,

no, automatic updates are disabled. Nevertheless all Phones were updated automatically, and we're now facing problems with this new firmware.

We have a big XML phonebook (around 7000 lines if you open the XML in an editor, with around 850 unique numbers in it). On some phones, dialing a specific internal extensions results in a crash of the phone application on the phone. The phone is unprovisioned afterwards and gets autoprovisioned after the app restarts (only the app crashes and restarts, not the complete phone). Changing the phonebook to a smaller phonebook solves this problem, but then the phonebook has only half of the numbers needed.

On the snom homepage is a bug entry for phonebooks > 1000, but this does not apply to our phones (D385) or the amout of unique numbers in the phonebook (~850):
https://service.snom.com/display/wiki/10.1.64.17+Release+Candidate

The firmware 10.1.64.17 is also listed as release candidate, which is classified as beta and not productive as you can see.

 

[JohnS_3CX]

Hi @mOrbo

The way the system works is

1. New firmware is downloaded and kept in a different folder than the one phones can see (to prevent automatic updating).

2. When the admin (not just you, anyone with admin access to the MC) decides to upgrade them, the firmware is moved into a folder that all the phones can see (so they will all auto-upgrade eventually when it comes to Snom phones)

Without step 2 being triggered manually, there is no way that this can happen, so it is indeed strange. You are also the only one who has reported this after the recent release of the new firmware. I'm wondering if somebody other than yourself has access and may have sent a firmware upgrade command from the management console. I'm also wondering if you use any DHCP Options on your network, that may possibly point the phones to where the firmware is (thus bypassing the method 3CX uses).

As for version 10.1.64.17, it does indeed appear as a RC candidate on their website, but Snom made the decision that it is ready for production for 3CX.

 

[mOrbo]

Hi John,

I asked all with access. They swear, they didn't click on "Phones -> Firmware -> OK". There was a red "1" under "Updates" in the Management Console, but after clicking on "Updates" there was nothing listed and the red "1" disappeared. One day later all Phones updated themselves over night to 10.1.64.17.
*edit* there are also no DHCP Options pointing to an upgrade path of the 3CX.

But the main problem right now is a potential memory handling bug in this firmware. I think you do have contact with the Snom support?

I can send you a video of the crashing phone app on the D385 phone. I also can send all logs needed.
After the crash and the autoreprovisioning there are also two Snom phones listed in the 3CX webclient of the affected extension, so this bug also affects the 3CX Webclient directly. One of the phones in the webclient disappears again after a while automatically.

 

[JohnS_3CX]

Understood. Firstly take a look in your PBX provisioning folder under:

..\Instance1\Data\Http\Interface\provisioning\xxxxxxxxxxxx\firmware\snom\

and confirm that the specific rom exists in that folder (name: snomD385-10.1.64.17-SIP-r.bin)

This would confirm that it was available to the phones as a first step.

 

[mOrbo]

Yes it's there, also under "/firmware_new/snom"

The timestamp of the file is exactly the same, in both pathes:

../Instance1/Data/Http/Interface/provisioning/xxx/firmware/snom# ls -la --time-style=full-iso
-rw-r--r-- 1 phonesystem phonesystem 36886176 2021-03-22 16:39:21.653840602 +0100 snomD385-10.1.64.17-SIP-r.bin

.../Instance1/Data/Http/Interface/provisioning/xxx/firmware_new/snom# ls -la --time-style=full-iso
-rw-r--r-- 1 phonesystem phonesystem 36886176 2021-03-22 16:39:21.653840602 +0100 snomD385-10.1.64.17-SIP-r.bin

 

[JohnS_3CX]

Ok, then it was definitely deployed at some point such that the phones would have updated them selves.

Meanwhile I also provisioned a D385 on a PBX with about 6000 unique contact entries
You will notice that the phone only provisioned the first 2000:

The phone was then able to process those without crashing.

So can you give some more details as per here?
https://www.3cx.com/community/threads/information-to-provide-when-requesting-help.67558/

 

[mOrbo]

Hi John,

are you using multiple entries per person i.e
- John Business 1234
- John Mobile 5678

or do you use only one entry with multiple number like this:
John
- Business 1234
- Mobile 5678

The second style is described here: https://service.snom.com/display/wiki/<tbook>,<phone-book>+tag
("multiple numbers per person are achieved by defining a Master-entry")

With the second config, we do have 450 entries (like in your screenshot above) with ~850 numbers.

The problem with the crash is a bit strange. You can't reproduce it on every extension, but if the problem exists at one specific extension, you can reproduce it every time. For example: If you call the extension 238 from extension 122, the app crashes. If you call 238 from extension 121 there is no problem. But 121 is crashing by calling 456 for example.
After changing the phonebook back to a smaller/simpler version, everything works again.

For the details:

• 3CX Version, in Dashboard: 16.0.8.9 / In License Settings: 16.0.9
• Server OS, Debian GNU/Linux 9 (stretch)
• Is the 3CX Server Hosted and where? OnPrem
• IP Phone Make/Model/Firmware. Snom D385 - Firmware 10.1.64.17
• Provisioning Method: Local
• Has the Firewall Checker passed: YES
• Are custom Phone Templates being used: YES and NO - Problem exists with both templates

 

[JohnS_3CX]

3CX does not support the second method you mentioned above. For each Contacts entry saved in 3CX, only the "mobile" field is populated in the phonebook that the phone downloads from your server.

So are you using some other method of populating the phonebook on the phone?

1. YES - If there are custom templates involved then we cannot help, you will have to contact Snom about this directly because the phone crashes in a method that 3CX does not support.

2. NO - If no custom templates are involved, then we expect each name entry on the phone to have only a single phone number, and in replicating this here I did not get the crashes.

 

[mOrbo]

Yes, we have to build our own phonebook, because this basic feature is missing in 3CX for years.

But it doesn't matter if we're using a custom template or simply replace the standard 3CX phonebook.

Why I try to fix the problem with you is, because you have a good B2B contact to snom. A support case from "a stupid customer" doesn't count very much I think. Besides that, they don't have any enduser support.

I also think, it is in the interest of 3CX to fix bugs in firmwares you deploy to customers.

 

[JohnS_3CX]

Why I try to fix the problem with you is, because you have a good B2B contact to snom. A support case from "a stupid customer" doesn't count very much I think. Besides that, they don't have any enduser support.

I would not call it a support case from "a stupid customer", you are a Snom customer and you are using a feature they provide (3CX aside) so I think you have a valid case to raise with them.

I also think, it is in the interest of 3CX to fix bugs in firmwares you deploy to customers.

Indeed this is true, but mainly for things that have to do with 3CX functionality. In this case, the phone does not crash with the way 3CX handles the phonebook, so it is out of our scope. If it was the 3CX phonebook then yes, we would definitely raise it with them.


I think it's worth reporting it to them directly, you may be onto something that they are already investigating, and having customer reports is helpful.

 

[mOrbo]

Ok, what a pity. I understand that, but a request from 3CX to Snom about fixing the firmware would have put more pressure on Snom. I will try to raise a case directly with the Snom Support.

Can you tell me a simple way to downgrade all our phones at once? With Firmware 10.1.54.16 everything was good. This would be a quick and dirty fix for us until Snom fixes the firmware.

Also it's still not clear why the upgrade started automatically.

 

[JohnS_3CX]

I would say it is not clear how the firmware got deployed at all, we haven't established if it was automatically or manually


As for the downgrade, I will PM you to help you.

 

[mOrbo]

At least all users with access swear they didn't click on "Phones -> Firmware -> OK".

You can't deploy Firmwares simply through "Settings -> Update", am I right?

Thanks in advanced for the PM.

 

[JohnS_3CX]

Correct, you can only download it from the update page but not deploy it.

To deploy it you must necessarily use the +Firmware button, otherwise the firmware will sit in a storage folder and the phones cannot see it yet.

 

[mOrbo]

Then I believe my co-admins. This happened as said on three installations. They could have updated one phone in one branch office, but not in all three locations.

The timestamps of the 10.1.64.17 firmware at all three locations differ by approximately 10 minutes. 10 minutes should be the time they need to open all 3 management consoles and do a search/download for updates, but not deploying it on all 3 locations.

Location 1:
2021-03-22 16:28:17.851004098 +0100 snomD385-10.1.64.17-SIP-r.bin

Loctaion 2:
2021-03-22 16:39:21.653840602 +0100 snomD385-10.1.64.17-SIP-r.bin

Location 3:
2021-03-22 17:02:57.050314812 +0100 snomD385-10.1.64.17-SIP-r.bin

Automatic Updates are disabled in all 3 Locations.

 

[simply7]

Well I think I found a bug. I have very strict firewall rules for outbound traffic. Basically the phones have full access to the cloud hosted 3CX instance. Which is great, until you notice that the Snom phones will not upgrade the firmware. After pushing an update and looking at the phone log, the download location in the D785 is not looking at the 3CX instance, it is trying to reach downloads.3cx.com. So now I have 200 phone failing their update. So I am wondering if your phones rebooted and grabbed the latest firmware from downloads.3cx.com directly and not from your 3CX instance. Has anyone else noticed this with regards to the firmware download location? I have a feeling that in the last template update something was changed.

Enterprise 16.0.8.9 Debian in AWS

 

[mOrbo]

@simply7

we are using an on-prem install. Our phones do not have any internet access. Even NAT is disabled for the VOIP VLAN-Subnet containing the Phones.

The Updatelink on the Phones is properly pointing to the local 3cx instance, so the update definitely came from the 3cx instance.

We're now doing a Firmware Rollback and disabling updates with a custom template with the parameter:
<update_policy perm="">never_update</update_policy>

Explanation from Snom:
"The "update_policy" field specifies the update policy for the phone. Valid values are:
"auto_update"==Update Without Prompting User,
"ask_for_update"==Prompt User To Allow Update,
"settings_only"==Retrieve Settings Only,
"never_update"==Do Not Retrieve Settings.
The provisioning template sets the value to "auto_update"==Update Without Prompting User."

With this setting we should be able to test Upgrades on single Phones, without an impact on all phones.