Q: 3CXPhoneSystemMC01 service stopped / WebAdmin login error

[fortech]

Hi, I've got a 'vanilla' 3cx instal (PBX_Express based, installed to OVH Public Cloud) which has been in service since ~6 months approx
It has been very smooth operating, and I have about 10 similar installs which all are ~same install / all work smoothly
They install updates automatically, generally work smoothly.

Today client sent me sad email, that all the phones were down. When I try to hit the web-admin portal login, I can see login page but it gives me a "SERVER ERROR" when I try to login with correct admin user-pass info

after banging my head on this for a while, I rebooted the Openstack VM / rescue mode / got in on VNC console / edited the shadow file of 3cx instance disk / allowed root login no pass on console / rebooted not in rescue mode / ie, rebooted 3cx normally / then in my vnc admin cloud console, logged in as root (Since I have no web admn, and no SSH access port open it seems by default, had to do this rescue mode VNC console fun). Anyhoo.

from there I issued command as root, "3CXServiceControl" and noted there was a service not running which sounded important, that being

3CXPhoneSystemMC01

so I kicked this one startup manually via command
systemctl start 3CXPhoneSystemMC01

and then after ~10 seconds it appeared to start

after that I was able to login normally to webUI Console. as admin user
From there the 'services' report on dashboard had one more service in red-sad state, the GATEWAY service. So that one I restarted manually via the WebUI

so, endgame for tonight,

this instance of 3cx appears to be back online
I can see client phone handsets are registered, I am guessing phones will work now. I can't test those handsets physically, office is on other side of town.
I will send update to client and say "yay, it works I think"
but I am still thinking to myself

what the hell happened

I did of course first try the classic IT guy fix, "reboot the server and see if that fixes it" - but that did not fix it. ie, I had to manually go in and poke this service started

I did find a hit in the forum discussing this, https://www.3cx.com/community/threads/3cxphonesystemmc01-service-stopped.49558/
and I am curious if anyone else is falling down this pit right now or if this sounds familiar, has known cause, other known reasons this sort of thing breaks suddenly with no known cause ?

lots of fun all around!

Happy days.

Tim

 

[TheodorosG_3CX]

This might require further investigation. As a first step ensure that the number of the extensions that are configured in the PBX and the hardware specifications of the machine hosting the PBX meet as per below guideline:
• https://www.3cx.com/docs/recommended-hardware-specifications-for-3cx/
Also check whether additional 3rd party services are not running on the machine causing the 3CX services interruption.

 

[Guillaume Bourgeois]

Hi, I've got a 'vanilla' 3cx instal (PBX_Express based, installed to OVH Public Cloud) which has been in service since ~6 months approx
It has been very smooth operating, and I have about 10 similar installs which all are ~same install / all work smoothly
They install updates automatically, generally work smoothly.

…

from there I issued command as root, "3CXServiceControl" and noted there was a service not running which sounded important, that being

3CXPhoneSystemMC01

so I kicked this one startup manually via command
systemctl start 3CXPhoneSystemMC01

and then after ~10 seconds it appeared to start

after that I was able to login normally to webUI Console. as admin user
From there the 'services' report on dashboard had one more service in red-sad state, the GATEWAY service. So that one I restarted manually via the WebUI

so, endgame for tonight,

this instance of 3cx appears to be back online
…

If you are one of the curious who has enabled the MULTICOMPANYMODE (MCM) feature in the custom parameters, Disable it !
( https://www.3cx.com/blog/releases/v18-update-4/
>> Other fixes and improvements )

This beta function (not recommended in production), I made the mistake of activating it in production, I encountered the same symptoms as you. I deactivated the setting, made a full backup and reinstalled the PBX with Backup.
Since then, this has never happened again.

A+

 

[fortech]

Hi, thanks for reply! I am quite positive I have not touched the custom settings area / enabled MCM multi-company mode. So in theory that is not an issue here.

 

[fortech]

This might require further investigation. As a first step ensure that the number of the extensions that are configured in the PBX and the hardware specifications of the machine hosting the PBX meet as per below guideline:
• https://www.3cx.com/docs/recommended-hardware-specifications-for-3cx/
Also check whether additional 3rd party services are not running on the machine causing the 3CX services interruption.

Hi, thank you for this reply. The deployment capacity of the VM running 3cx here is pretty standard. I have >~10 different clients on similar spec config and it is ok for all of them so far / I have never seen this issue elsewhere. I believe VM specs are (approx) Single-core Xeon / 2gb ram / 10gig disk / Openstack KVM OVH Public cloud vanilla deploy using recommended VM template as per the PBX_Express auto-config-deploy toolset. This particular 3cx instance is servicing a pretty small client site (14 extensions on the system, 6 of them lit up and the rest don't have active 3cx client device most of the time) 3cx is pro-annual-4SC version and I would be surprised if they have more than 2 concurrent calls active ever. There should be no extra stuff on this debian/3cx instance - I did zero customization, only did vanilla 3cx setup via the 3cx web admin GUI in terms of (a) setup sip trunk (b) setup users/EXT (c) setup a basic IVR Greet structure for the one office DID (d) that is about it. -Tim

 

[Guillaume Bourgeois]

Hi, thank you for this reply. The deployment capacity of the VM running 3cx here is pretty standard. I have >~10 different clients on similar spec config and it is ok for all of them so far / I have never seen this issue elsewhere. I believe VM specs are (approx) Single-core Xeon / 2gb ram / 10gig disk / Openstack KVM OVH Public cloud vanilla deploy using recommended VM template as per the PBX_Express auto-config-deploy toolset. This particular 3cx instance is servicing a pretty small client site (14 extensions on the system, 6 of them lit up and the rest don't have active 3cx client device most of the time) 3cx is pro-annual-4SC version and I would be surprised if they have more than 2 concurrent calls active ever. There should be no extra stuff on this debian/3cx instance - I did zero customization, only did vanilla 3cx setup via the 3cx web admin GUI in terms of (a) setup sip trunk (b) setup users/EXT (c) setup a basic IVR Greet structure for the one office DID (d) that is about it. -Tim

Are you able to reproduce the issue? If yes, activate VERBOSE mode (Activity logs) and then reproduce the problem.. then generate a Support Info package.
Open a 3CX Support Ticket for investigation. Please don't put the supportinfo package here, it contains a lot of information. It’s almost the equivalent of a system backup with logs.


Good luck !

 

[fortech]

Are you able to reproduce the issue? If yes, activate VERBOSE mode (Activity logs) and then reproduce the problem.. then generate a Support Info package.
Open a 3CX Support Ticket for investigation. Please don't put the supportinfo package here, it contains a lot of information. It’s almost the equivalent of a system backup with logs.


Good luck !

Thank you - sounds like a good suggestion. I reproduced the issue yesterday (I guess you can call it that?) in that I rebooted the VM 2 times in a 1 hour period and the problem was not resolved after a reboot / management UI was busted. And then I manually fixed management UI by starting the one service as per my initial thread posting about VNC Console access / root login etc. Thanks for tip re: Do not post this file here! and if I get a reply / and progress after this I will be sure to update thread so others might see the outcome of this little adventure. --Tim

 

[fortech]

Update on thread in case it is helpful to others in future.

I did reach out to 3cx support via ticket, and I provided upload of the debug logs file. Initial response from support was this:
-- my VM does not meet recommended spec requirements for current-latest version of 3cx
-- please upgrade VM capacity to meet specs
-- test after that, and update the ticket.
-- for reference, this VM was deployed using the 3cx automated "PBX Exporess" deploy tool, using the VM size which had been recommended/automated in that tool. Which was basically an entry level (OVH Public Cloud) instance with 1x vCPU, 2gb ram.
-- I waited until after-hours, did a VM resize on OVH Cloud Admin Panel, bumped it up to a larger VM size with more resources - 2vCPU, 4gb ram.
-- after doing this, and restarting the 3cx pbx, everything now works correctly again. No services fail to auto-start on reboot. The problem appears resolved.

So, my only remaining mystery is that - I still don't really understand what was going on. The fact I could manually start services via SSH CLI session / but they did not auto-start at boot seems a bit odd. Possibly the RAM requirements in this particular config were such that the services auto-starting did not like this host config. I've still got >approx a dozen other clients who are on the old (Smaller, default PBX Express) Deploy config and the all remain operational / no drama / no problems when reboted. This client is among the smallest ones I've got deployed - not largest - so - anyoo.

I guess end of the day, 3cx have changed the recommended minimum specs needed to run the PBX Server Debian instance to demand a bit more CPU and RAM than used to be the case. And they will support instances which have recommended specs.
and. In future. I have feeling that 3cx is moving towards a model where service providers might be encouraged to use one instance of 3cx with multiple clients hosted on it / with each instance being 'virtual/independant standalone' but in reality it would be "one physical server instance". Arguably this would be fine with me - I could just spin up one beefier 3cx instance on the new platform, and then presumably import old-prior sites using backups (hopefully?) to bring them in. Then depreciate the old small standalone instances. Leaving me with fewer VM to manage in paralllel. (Ultimately the current model, with multiple small VM - was working fine, and really management was pretty simple / hands off. But anyhoo).

So. that is end of this thread for now. Happy Monday everyone...!

Tim