Voicemail Unregistered - Auth Failed?

[Evolute IT]

Hi,

We are currently getting a few of these a week... Voicemail (999) going Unregistered in the System Extensions. When looking at the logs, we see odd stuff: Auth Failed. Apparently, the server forgot his own password or something like that.

Restarting the IVR Service alone didn't do it. We had to restart SIP Server, IVR Server and Config Server to get the VM 999 to re-register.

Attached are screenshots of the logs when failed and when succeeded after service restart.

 

[Saqqara]

Please provide info on the setup - https://www.3cx.com/community/threads/information-to-provide-when-requesting-help.67558/

 

[Evolute IT]

Setup:

- 3CX v18 U3, latest build (hotfix), installed via the ISO (Debian 10)
- Hosted in datacenter on KVM
- Phones are DesktopApp (latest as well) / Via HTTPS (firewall checker passes)
- No Custom Templates
- System is fully up-to-date (also autoupdates)

This is a brand new system that we haven't even delivered to the customer yet. This also happens on another customer's system where they get MANY calls. We thought it might be RAM but it wasn't.

The fact that this gives out Auth errors is very mindboggling.

 

[SteveITS]

I seem to recall seeing another post about this... I thought recently but this one is from January:
https://www.3cx.com/community/threads/voicemail-service-unregistered.87453/
Have not noticed it myself. Have you restarted the entire server?

 

[NickD_3CX]

@ConceptsWeb
Are you getting any notifications about stopped services?
This could happen if the SIP Server stops abruptly without the IVR service stopping and restarts immediately.

The IVR server would try again with the same Auth but the SIP Server after the restart will have "forgotten", so for sure the first reply would be to challenge the REGISTER request.

 

[Evolute IT]

@ConceptsWeb
Are you getting any notifications about stopped services?
This could happen if the SIP Server stops abruptly without the IVR service stopping and restarts immediately.

The IVR server would try again with the same Auth but the SIP Server after the restart will have "forgotten", so for sure the first reply would be to challenge the REGISTER request.

Sorry for the delay, we tried a few things (made sure NTP is there and also allocated more CPU and RAM, but it wasn't really the issue.)

No notifications, nothing. It happens randomly during the day, nothing gets restarted (we checked logs and we have notifications enabled for services not running.)

Here's what we get:

 

[Evolute IT]

I seem to recall seeing another post about this... I thought recently but this one is from January:
https://www.3cx.com/community/threads/voicemail-service-unregistered.87453/
Have not noticed it myself. Have you restarted the entire server?

Restarting the server is not an option. It just has to NOT crash lol

Same goes for services.

 

[SteveITS]

It just has to NOT crash lol

Hmm, try this plugin:
if (goingtocrash()) {
dont;
}

I actually meant, "does it happen again after a restart" not "restart every time it happens."

 

[Evolute IT]

Hmm, try this plugin:
if (goingtocrash()) {
dont;
}

I actually meant, "does it happen again after a restart" not "restart every time it happens."

It happens anytime and yes even if the server was rebooted the day before. It's just random and there's nothing we could see that could be the trigger.

 

[NickD_3CX]

I just noticed something a little unusual in the logs:




Are you actually using these SIP Ports on these installations?

Also I just want to point out that there is no talk about a service crashing, this is just an Endpoint (yes, VMail is still an "Endpoint"...) unregistering.

I'm not saying that using custom ports is prohibited or anything like that, just trying to find a common denominator.

What I'd do next just to get a better picture is start a tshark on one of the servers where this happens fairly frequently, probably with a capture filter with something like this:

sudo mkdir /home/vmail_captures/
sudo tshark -F libpcap -i any -f "host 127.0.0.1 and udp port 5483" -b filesize:20480 -w /home/vmail_captures/vmail.pcap

This will capture all the SIP traffic originating from the IVR Service, then to isolate the VMail only,

Then once it happens open the capture file(s) and to check only for the VMail menu, I'd probably use a Wireshark Display filter sip.CSeq.method==REGISTER and frame contains "Voice Mail Menu".

 

[Evolute IT]

I just noticed something a little unusual in the logs:
View attachment 29356

View attachment 29357

Are you actually using these SIP Ports on these installations?

Also I just want to point out that there is no talk about a service crashing, this is just an Endpoint (yes, VMail is still an "Endpoint"...) unregistering.

I'm not saying that using custom ports is prohibited or anything like that, just trying to find a common denominator.

What I'd do next just to get a better picture is start a tshark on one of the servers where this happens fairly frequently, probably with a capture filter with something like this:

sudo mkdir /home/vmail_captures/
sudo tshark -F libpcap -i any -f "host 127.0.0.1 and udp port 5483" -b filesize:20480 -w /home/vmail_captures/vmail.pcap

This will capture all the SIP traffic originating from the IVR Service, then to isolate the VMail only,

Then once it happens open the capture file(s) and to check only for the VMail menu, I'd probably use a Wireshark Display filter sip.CSeq.method==REGISTER and frame contains "Voice Mail Menu".

Yes we do use special ports simply to avoid hackers and also offer more security. All ports are random like that (HTTP(S), SIP and Tunnel.)

I will check if we can do the capture so that it runs continuously in the background. Not sure what it will give more than the activity log.

But yes, the endpoint is basically unregistering for some reason but it's not indicated or easy to figure out.

 

[NickD_3CX]

Not sure what it will give more than the activity log.

Might be easier to see when de-registrations happen and what exactly it is doing when it tries to re-register after.

If/when this happens again, don't restart services immediately, if possible leave it for an hour or so so that it attempts a few times by itself, just to get a better idea where it goes wrong.

 

[Evolute IT]

Might be easier to see when de-registrations happen and what exactly it is doing when it tries to re-register after.

If/when this happens again, don't restart services immediately, if possible leave it for an hour or so so that it attempts a few times by itself, just to get a better idea where it goes wrong.

We left one for 7 hours yesterday before restarting the services. That would've been perfect.