No access to Web Management Console since last update

[levbe]

Since the last update (SP3) on a Debian installation, i can't access the Management Console.
The web login page says : Server Error.

When i try to restart the MC service, i get the following error :

Job for 3CXPhoneSystemMC01.service failed because a fatal signal was delivered causing the control process to dump core.

See "systemctl status 3CXPhoneSystemMC01.service" and "journalctl -xe" for details.
root@minipc01-3cx:~# systemctl status 3CXPhoneSystemMC01.service
● 3CXPhoneSystemMC01.service - 3CX PhoneSystem 01 Management Console
   Loaded: loaded (/lib/systemd/system/3CXPhoneSystemMC01.service; enabled; vendor preset: enabled)
   Active: failed (Result: core-dump) since Tue 2018-02-13 18:59:19 CET; 57s ago
  Process: 962 ExecStopPost=/bin/rm -f /var/run/phonesystem/management_console01.socket (code=exited, status=0/SUCCESS)
  Process: 960 ExecStopPost=/bin/rm -f /var/run/phonesystem/3CXPhoneSystemMC01.pid (code=exited, status=0/SUCCESS)
  Process: 935 ExecStart=/usr/lib/3cxpbx/ManagementConsoleJS --pid /var/run/phonesystem/3CXPhoneSystemMC01.pid -p /var/lib/3cxpbx/Instance1/Data/Http/webroot (code=dumped, s
ignal=ILL)
 Main PID: 935 (code=dumped, signal=ILL)

Does it sounds familiar to someone ?

 

[Ramped]

Good find @manfer

Overnight 3cx have been on our system making a few changes and doing a bit of testing. I'm not 100% sure what has been done as yet, though I do have access to the Management console again! I will await an update form support and pass on any relevant info!

Starting to look like a resolution is in sight!

 

[levbe]

All that sounds very very good.
Thanks #manfer for your investigations. I'll try to follow the same way on my system (as its still a test environment for some days yet).

 

[complex1]

Hi,

I experience the same issue…
After some time I get a "502 bad gateway" and then receive an email notification “Following service(s) were stopped recently: 3CXPhoneSystemMC01”
All extension stop working and when logging into the web interface I get "Server Error".
To get the system back on line I have to reboot the intel NUC.

3CX v15.5 SP3 on Debian Stretch 9.3

 

[YiannisH_3CX]

Thank you all for reporting and providing assistance in troubleshooting this. We are currently looking into the issue and i will you updated with any new developments.

 

[emo89]

Thank you all for reporting and providing assistance in troubleshooting this. We are currently looking into the issue and i will you updated with any new developments.

Thank you Yannis. Do you have a date when an resolution is found? Because I had to change a ringgroup in our customers pbx. When the resolution will take more then 2-3 days, i have to install 3cx again on windows to make this changes. Thank you

 

[JCLloyd]

It looks like there is another thread with a different issue seemingly related to SP3. So far, I am not affected by either.

If my system can give insight into why the other systems are having issues, contact me. I will do what I can to assist with the hardware and software configurations in my environment.​

Thank you all for reporting and providing assistance in troubleshooting this. We are currently looking into the issue and i will you updated with any new developments.

 

[YiannisH_3CX]

Thank you Yannis. Do you have a date when an resolution is found? Because I had to change a ringgroup in our customers pbx. When the resolution will take more then 2-3 days, i have to install 3cx again on windows to make this changes. Thank you

Unfortunately as this is currently under investigation we cannot know when a solution will be found and what the solution might be so i cannot give you an answer at this moment.

 

[YiannisH_3CX]

If my system can give insight into why the other systems are having issues, contact me. I will do what I can to assist with the hardware and software configurations in my environment.

Thank you for the offer @JCLloyd

 

[YiannisH_3CX]

Is there anyone here experiencing the issue that does not run a production system and can give us access so we can further troubleshoot the issue? The more systems we can access the easiest it is to find the cause. We have tried reproducing the issue in house but we were not able to.
Please send me a p.m. so we can arrange access to these systems.

 

[levbe]

PM sent !

 

[norekhov]

We have identified the issue for MiniPC. Looks like SP3 will have issues on special CPU's without AVX extensions. Here you will need to use SP2.
We're waiting for this fix to be merged and released in NETCore: https://github.com/dotnet/coreclr/issues/16052
Thanks for your help in investigating!

Please check if you're not affected here: https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2

 

[levbe]

Thanks a lot... i have "Apollo Lake J3455 CPU" in my MiniPC, which does not support AVX, so that's it.

 

[Marari]

Seems to be a random issue on my Google Cloud hosted PBX as well (Debian / g1-small (1 vCPU, 1.7 GB memory)

 

[emo89]

Thanks a lot. How we can install 3CX with SP2 ? Can you give us a link to an debian ISO with SP2 ?

 

[maumee]

So, what are we supposed to do to fix this??

I just tried calling support only to receive a message "There are no available agents in the queue at this moment" and hung up on.

As a partner, I'm shocked to get this message, and it's even worse that my client's phone system is broken. I cannot fix their broken extensions. It's making us look really bad at the moment.

 

[levbe]

For anyone else having major issues with Update 3 here is how you can downgrade to 2 (at your own risk)-

1. Backup your system (3cx doesn't recommend restoring an Update 3 backup to a lower update)
2. SSH into your server and run the following commands
3. apt-get remove 3cxpbx
4. apt-get autoclean
5. apt-get install libcurl3=7.38.0-4+deb8u5
6. apt-get install 3cxpbx=15.5.6354.2
7. apt-mark hold 3cxpbx
   
8. /usr/sbin/3CXWizard --cleanup (if you want to)

 

[JCLloyd]

Thank you for figuring this out, documenting and sharing it with us!

 

[emo89]

thank you bltech and levbe. the comands are worked well and I can use 3CX for now. Let´s see if this will fixed in the next weeks for CPU´s without AVX extention

 

[Ramped]

As per the commands provided, we have also rolled back without issue.

FYI:
Interestingly we had also been chasing a call quality issue, whereby audio gradually become degraded whilst on a call. As we had a new internet connection installed the day we upgraded 3CX to SP3, we initially figured it was an internet config/QOS issue.

Since the downgrade, everything is back to normal!

 

[emo89]

any news ?
because on github the thread is closed.