Server Error - 3CX Management Console #6480

[dacosta]

Hi guys, i know there a lot of threads, but most of them if not all just states you need to restart 3CX's service and/or is associated to Hosted 3CX, which is not my case, i run a on-premises Windows install.



In my recent incident (last Thursday), the service was not stopped on the Windows Server. But restarting its service (3CX Management Console - also 3CX Gateway which is service dependant) got me back logging as well as restablishing 3CX presence and agenda.

Since the service was running, i wasn't able (although i did it anyway) to find any event log errors, 3CX console logs doesn't such much too, our port probe with PRTG did not triggered (TCP 5000/TCP 5001), we didn't receive 3CX notification email abouts its services, download support file to look into with 3CX log viewer it also didn't help, since there wasn't info from incident time - i went to collect it later.

Does someone know what does it trigger this service malfuncion? How to solve and/or proper monitor it?

Associated links that i dug it through before posting this.
https://www.3cx.com/community/tags/server-error/
https://www.3cx.com/community/threads/management-console-login-server-error.76111/
https://www.3cx.com/community/threads/server-error-while-logging-in-to-management-console.86082/
https://www.3cx.com/community/threads/3cx-management-console-server-error.81917/
https://www.3cx.com/community/threads/server-error-when-trying-to-login.80478/
https://www.3cx.com/community/threads/server-error-on-login-page.64783/
https://www.3cx.com/community/threads/unable-to-log-in-server-error.62236/
https://www.3cx.com/community/threads/3cx-v15-server-error-or-bad-gateway.54763/
https://www.3cx.com/community/threads/server-error-on-3cx-management-console.53870/

Regards,
Rafael da Costa

 

[VasilisV_3CX]

Hi @dacosta,

Do you have an offsite backup saved? If not, I'd suggest moving the backup off of the machine.

Have you ensured that your nginx service is running? Can you restart that as well?

Note that this may have been caused by a hardware failure, most notably a RAM failure. I'd have a look at this option as well.

 

[dacosta]

Hi Vasilis,

i do have offsite backup, we use an external FTP server. Just making things clear, as soon as i restarted 3CX Management Console service - which was running, i was able to login to the console again.

I didn't pay attention to ngnix service, but since, it was showing up the login page, i believe the service was running too. Also, i did look through service stop logs via powershell and event viewer and i didn't get much info neither info related to ngnix service.

I don't know how to troubleshoot RAM failure, but i'll research it. If you have any useful information related, i'd be very pleased. Thank you.



Regards,
Rafael da Costa

 

[VasilisV_3CX]

Hi @dacosta,

To correct what I said earlier, it's more likely to be a lack of RAM available to the system more than a RAM failure.

i was able to login to the console again.

Restarting the management console allows you to log in, are you kicked out or does this only happen the next time you try to log in?

 

[dacosta]

Vasilis,

Restarting the management console allows you to log in - YES
are you kicked out - Not sure, i wasn't logged on the moment of failure.
does this only happen the next time you try to log in? Most likely, but i'm uncertain.

I'll try to monitor available RAM. Could 3CX trigger some e-mail notification to advise this? As it does with other failing services?

Regards,
Rafael da Costa

 

[NickD_3CX]

To chime in here as well, could it be that your Windows Server restarted to install any updates?
Sometimes when 3CX is installed on Windows, the 3CX Services try to start faster than Windows, so the solution to this is to set your 3CX services to "Delayed Start":

 

[dacosta]

Hi, Vasilis, checking our WMI/PRTG sensor for memory, the same goes low as 60%, we have a 6GB Core I5 computer for this PABX.

The graphis below are split because we migrated our monitoring service recently.


Nick, the computer did not rebooted recently. The last reboot was on March, 8th and the incident was reported on March 10th. It could be no one noted before, but without management console properly running, there wouldn't be agenda, presence and softphones properly working.


Regarding delayed start, i'm suspicious about it too, being a point of failure, so i did set 3CX Management Console service to delay its start. I'll get to see how it goes from now on.

Regards,
Rafael da Costa

 

[dacosta]

P.S.: This environment has a bunch of grandstream ip phones, it may be a webservice failure not be noted.

Since phone features were working properly at the time of the incident. Also two days before, i granted an extension right to perform operations. On the following day, March 09th no one complained until March 10th morning, local IT reached us with this issue.



Regards,
Rafael da Costa

 

[NickD_3CX]

This shouldn't have anything to do with it, so please monitor it and let us know.

If you find your 3CX Server in this state again, before taking any actions, please go to Windows Service and take a screenshot of what services are started and what is stopped.