Issues with t46s firmware - no service

[cooper.simpson]

We have 30 T46 phones. Thursday, everything working 66.84.0.35 which was saying up to date. Friday, come to work, 7 phones dead, saying no service, rest are fine, say unsupported, but no problem I push an update firmware and all is hunky dory.

What am I to do with 1400$ of bricked hardware now for the other 7 phones?

I have factory reset, manually updated firmware, they boot up, provision (I login to web GUI and see they are provisioned) but just say reg fail. Date/Time is all correct, account creds it pulls down from 3CX is correct.

What I take issue with the most of all of this is I have my 3CX set to not auto update, so why in the hell did it do something on its own seemingly late Thursday or on Friday?

 

[JohnS_3CX]

Hi Cooper,

So to better understand the situation, 7 phones stopped working overnight, while the rest were ok both before and after the firmware upgrade?

 

[cooper.simpson]

Indeed. @JohnS_3CX

Working phones both firmware .84.0.35 and 84.0.95
Not working phones both firmwares as well.

The firmwares were never out of date and updating them manually does not fix the problem with the firmware provided on the 3CX website.

All the provisioning makes it to the phone no problem and its sitting there attempting to register. This is debian 3CX. Is there somewhere in debian perhaps its blocking these phones based on their mac addresses? Need to try everything.

 

[AWS2P]

Are your phones same Lan as PBX ?

 

[cooper.simpson]

Are your phones same Lan as PBX ?

Yes.

 

[JohnS_3CX]

Ok so it looks like the firmware is not related to the case.

What might be happening is that the phones got blacklisted. That should be easy to see from your management console. In case they are, you can remove their IPs manually.

Blacklisted phones provision but will not be allowed to register.

 

[cooper.simpson]

Ok so it looks like the firmware is not related to the case.

What might be happening is that the phones got blacklisted. That should be easy to see from your management console. In case they are, you can remove their IPs manually.

Blacklisted phones provision but will not be allowed to register.

IP Blacklist? Empty. Where can I see blacklisted phones?

 

[JohnS_3CX]

Ok the list is clear, lets look at the next part:

Their attempts will be visible in the event log on your Dashboard, whether they try to register or got blocked and the blacklist rule now expired. You can click on the word "Event Log" to launch the whole log and see the part between Thursday and Friday

 

[cooper.simpson]

There are no entries in the log for these devices, even if I reboot them nothing comes to the logs.

Here is the log, at 9:01 I added a device to this users extension. I then rebooted it, and it provisioned as I can see on the phone it has Extension 35 on it, but not registering. This is the very top of the log, its now 9:11 and no entries are made since 9:03

10/22/2019 9:03:59 AM - Endpoint Extn:35 has refreshed contact <sip:[email protected]:5483/UDP>

10/22/2019 9:01:56 AM - Extn:35: Synch devices to dndOFF

10/22/2019 9:01:56 AM - Extn:35: DND change request to dndOFF is ignored - same state is charged already

10/22/2019 9:01:56 AM - Extn:35: DND change request to dndOFF is ignored - same state is charged already

10/22/2019 9:01:56 AM - Extn:35: DND change request to dndOFF is ignored - same state is charged already

10/22/2019 9:01:56 AM - Extn:35: Starting DND change transaction to dndOFF requested from configuration

10/22/2019 9:01:56 AM - Settings of Extn:35 has been updated

10/22/2019 9:01:56 AM - Extn:35: Profile is set to dndOFF

10/22/2019 9:01:56 AM - OverrideExt.35 has inactive profile override

10/22/2019 9:01:56 AM - Settings of Extn:35 has been updated

10/22/2019 9:01:56 AM - Extn:35: Profile is set to dndOFF

10/22/2019 9:01:56 AM - OverrideExt.35 has inactive profile override

10/22/2019 9:01:56 AM - Settings of Extn:35 has been updated

10/22/2019 9:01:56 AM - Extn:35: Profile is set to dndOFF

10/22/2019 9:01:56 AM - OverrideExt.35 has inactive profile override

10/22/2019 9:01:56 AM - Settings of Extn:35 has been updated

10/22/2019 9:01:56 AM - Extn:35: Profile is set to dndOFF

10/22/2019 9:01:56 AM - OverrideExt.35 has inactive profile override

 

[JohnS_3CX]

Let's try to provision one of the problematic phones again by this method please:

1) First, plug 1 problematic phone, into the LAN cable of one of the good phones (as a test to eliminate any network related issue)

2) Then, if it does not register still, we will find the extension of that phone and remove the phone from the extension completely and click ok to save

3) We will reset the phone now (hold OK button down for a few seconds until you see reset prompt)

4) After the reset, the phone should appear as new in your Phones tab. Select it and assign it to your extension. Ensure you are not using a custom template for this.

After provisioning is complete, and while the phone is still in the LAN cable that belongs to a good phone, lets see if it can register.

 

[cooper.simpson]

Let's try to provision one of the problematic phones again by this method please:

1) First, plug 1 problematic phone, into the LAN cable of one of the good phones (as a test to eliminate any network related issue)

2) Then, if it does not register still, we will find the extension of that phone and remove the phone from the extension completely and click ok to save

3) We will reset the phone now (hold OK button down for a few seconds until you see reset prompt)

4) After the reset, the phone should appear as new in your Phones tab. Select it and assign it to your extension. Ensure you are not using a custom template for this.

After provisioning is complete, and while the phone is still in the LAN cable that belongs to a good phone, lets see if it can register.

I am doing this now, but I have done this process every time I have tried fixing them already.
You say the phone should appear in the phones tab? I have never seen that behavior before on an unprovisioned phone.

Anyway, factory resetting now, again, and will report back in five minutes with the news.

 

[AWS2P]

if you never see a new or factory reset phone added to your LAN appear in bold in MC , you have a multicast problem.

Do you use VLans , how is your LAN topology?

 

[JohnS_3CX]

Ok, if a freshly reset phone does not appear on the phones tab as a new device, this is an indication that there is something happening on the network or the PBX network config.

By default, when you reset a phone it sends a SUBSCRIBE on the multicast IP that your PBX normally listens to, and it appears on bold on your phones section:


If this is not happening, then there is either an issue with your network setup, or the PBX NIC setup.

That can be the key that could explain why some phones work and some don't, regardless of FW version.

 

[cooper.simpson]

I have sent this multicast thing over to some networking folks who can look into that. But this has been our behavior for the last year, never had an issue previously. So lets continue looking at other things in the meantime.

 

[JohnS_3CX]

Another test that can be very revealing of network issues, is to run a capture on your PBX via the Activity Log menu and then turn on the problematic phone (must be a provisioned phone).

If it tries to register on the PBX the capture should show it. If no registration attempt arrives then most likely the network communication between phone<->PBX is broken somewhere. This can be due to misconfigured network equipment for example but at least you can know with a high degree of certainty that the registration attempt did not reach your PBX

 

[cooper.simpson]

I have all my phones up and running right now. I powered them all off for 30 minutes and rebooted them. Gotta be some stale cache of some sort in the PBX.

Will address the multicast issue separately.

 

[AWS2P]

Now new investigations on muticast

 

[cooper.simpson]

Not going to proceed with Multicast. Multicast Forwarding is blocked on the firewall and we simply will continue using Opt66 I am told.


So back to the original issue, any idea what could've cached perhaps but cleared after turning the phones off for a few minutes? If you look at my logs its trying to send or communicate with 127.0.0.1:5*** yet on a properly registered phone it shows the phones proper IP. odd. after the power off for several minutes they registered...

 

[BrenttG]

Sounds like you could have some internal DNS issues going on too, but there are a lot of things that it could be, but its definitely not firmware, and its 99% most likely not the PBX itself. VLANs and/or auto voice vlan on some ports, or on a particular switch that was recently worked on could cause this.

Has any server, dns, or switch work been done in this same time period?

 

[JohnS_3CX]

Hi Cooper,

I don't know how your network environment is set up so I cannot say why the 30 mins made a difference, but I can say that the DNS comment does not make much sense..

Local phones would not normally rely on DNS because they speak with the PBX and each other directly via IP addresses. To me it sounds more like the firewall is doing something that may be interfering, but again, I don't know what your network environment is like so this is speculation on my part. I think you need to see what the network is doing more than anything else, and the captures method I mentioned earlier is a good place to start.




Note: The 127.0.0.1 you see in logs is normal, it appears on any healthy PBXs when they refer to themselves (ie the IVR traffic, tunnel traffic etc). Again unrelated to the issue you are facing in this case.