3CX Debian Services Won't Start

[Peter Richardson]

Hi All,

Today I had a client call me saying that their phone system was down. Upon investigation, I couldn't log into the web interface, I just got "server error". Very informative. So I restarted the VM. Same result. I restarted twice more. Same result. I was able to SSH into Debian and view the services with the ps -aux |more command and I noticed that none of the 3CX services were running. So I looked around for where it might be and I found it in /usr/sbin - 3CXStartServices and launched it.

Why did 3CX stop?
Why didn't 3CX services start at login?
Why isn't there anything in the event log about why it crashed?

On a side note - Android app - why does it re-register every time I switch to the app? Doesn't it run in the background? It has to make a connection to the server every time I switch to another app. And it has to wait to download the call history EVERY time. Doesn't it have a memory whatsoever?

 

[Peter Richardson]

Anyone?

 

[YiannisH_3CX]

Today I had a client call me saying that their phone system was down. Upon investigation, I couldn't log into the web interface, I just got "server error". Very informative. So I restarted the VM. Same result. I restarted twice more. Same result. I was able to SSH into Debian and view the services with the ps -aux |more command and I noticed that none of the 3CX services were running. So I looked around for where it might be and I found it in /usr/sbin - 3CXStartServices and launched it.

Why did 3CX stop?
Why didn't 3CX services start at login?
Why isn't there anything in the event log about why it crashed?

What PBX version are you running and on what Debian version? The most common reason for services to stop is resources. Make sure that you have sufficient disk space, memory and CPU power available on the machine. Also make sure that the machine was not rebooted for some reason.

Usually 3CX services might not start after a reboot because when the services try to start the network is still unavailable so the services fail to start. That would explain why you were able to start them manually.

You could possibly find more information regarding the crush in the Linux syslog. You can access it through the management console by navigating to Dashboard / Activity log / Logs.

 

[YiannisH_3CX]

On a side note - Android app - why does it re-register every time I switch to the app? Doesn't it run in the background? It has to make a connection to the server every time I switch to another app. And it has to wait to download the call history EVERY time. Doesn't it have a memory whatsoever?

Please note that the 3CX deregisters from the PBX when you close the app or put it in the background. You will notice that the app is visible under the phones tab of the management console while it is open which means it is registered and it will not be visible in the phones tab once you close it and the registration expires.
Once the app is in the background it stops registering to the PBX and it is waiting for Push messages to wake up. This is done to save battery as the app is not constantly trying to connect to the PBX and retrieve information. So when you start the app it will always need to re-register and establish a connection to the PBX.

 

[Peter Richardson]

Thanks @YiannisH_3CX

The VM has plenty of CPU power, RAM and HDD space available, nothing has changed there.

Running on Debian 8.

Make sure the machine wasn't rebooted? The PBX should be able to recover from being rebooted. I don't think you'd be able to sell it if it didn't.

The network was not down because I was able to log into the VMWare host remotely, so definitely not a network issue. Even if it was a network issue, it should be able to recover from a network outage, or else you shouldn't be selling it. Networks go down from time to time - there is no such thing as 100.00% uptime.

Which log file should I be reading? There are many. I have looked at a couple, and I have found error messages in 3cxManagementConsole.log such as:

ProductUpdater.UpdateException: WARNINGS.UNABLE_REACH_UPDATES_SERVER
   at ProductUpdater.Services.DefaultUpdateDefinitionProvider.<Download>d__3.MoveNext()

2018/08/05 10:28:39.182|14192|0003|Inf|PRV: Requested file '/provisioning/OBFUSCATED/OBFUSCATED.cfg' will be generated
2018/08/05 10:28:40.772|14192|0003|Inf|PRV: Requested file '/provisioning/OBFUSCATED/logo/yealinkt22.dob' cannot be generated
2018/08/05 10:28:40.872|14192|0026|Inf|PRV: Requested file '/provisioning/OBFUSCATED/Directory_setting.xml' cannot be generated
2018/08/05 10:28:41.202|14192|0003|Inf|PRV: Requested file '/provisioning/OBFUSCATED/firmware/yealink//T41S.rom' cannot be generated
2018/08/05 10:28:41.203|14192|0003|Inf|PRV: Requested file '/provisioning/OBFUSCATED/firmware/yealink//T41S.rom' is not found in file system also
2018/08/05 10:28:44.299|14192|0038|Inf|PRV: Requested file '/provisioning/OBFUSCATED/firmware/yealink//YL0000000000.rom' cannot be generated
2018/08/05 10:28:44.299|14192|0038|Inf|PRV: Requested file '/provisioning/OBFUSCATED/firmware/yealink//YL0000000000.rom' is not found in file system also

2018/08/05 18:53:39.245|645|0007|Err|VMMonitor: directory not exists: /var/lib/3cxpbx/Instance1/Data/Ivr/Voicemail/Extensions/100
2018/08/05 18:54:18.320|645|0007|Err|VMMonitor: directory not exists: /var/lib/3cxpbx/Instance1/Data/Ivr/Voicemail/Extensions/103
2018/08/05 18:54:21.872|645|0007|Err|VMMonitor: directory not exists: /var/lib/3cxpbx/Instance1/Data/Ivr/Voicemail/Extensions/102
2018/08/05 19:05:50.133|645|0014|Inf|Making GoogleDrive request for type application/vnd.google-apps.folder
2018/08/05 19:05:50.735|645|0014|Inf|Making GoogleDrive request for type application/x-zip-compressed
2018/08/05 20:54:34.404|645|0022|Inf|PRV: Starting to initialize...
2018/08/05 20:54:34.436|645|0022|Inf|PRV: 13 provisioning files successfully initialized
2018/08/05 20:54:34.439|645|0022|Inf|PRV: Requested file '/provisioning/OBFUSCATED/yealink_phonebook.xml' cannot be generated
2018/08/05 20:54:39.524|645|0013|Inf|PRV: Requested file '/provisioning/OBFUSCATED/yealink_phonebook.xml' cannot be generated
2018/08/05 20:54:58.516|645|0013|Inf|PRV: Requested file '/provisioning/OBFUSCATED/yealink_phonebook.xml' cannot be generated
2018/08/05 20:55:49.159|645|0014|Inf|PRV: Requested file '/provisioning/OBFUSCATED/yealink_phonebook.xml' cannot be generated

There are a lot of different errors here, any idea what they are caused by? None of them seem to give any clues (at least to me) as to why the services couldn't start, but there are many log files, so I might be looking in the wrong ones.  There wasn't much in 3cxSystemServices.log on that day, just looks like someone mis-dialled a few times:

2018/08/05 18:56:21.159|647|0007|Erro|Call type is invalid or not specified Unknown
2018/08/05 18:56:21.162|647|0007|Erro|Call type is invalid or not specified Unknown
2018/08/05 18:56:21.163|647|0007|Erro|Call type is invalid or not specified Unknown
2018/08/05 18:56:21.163|647|0007|Erro|Call type is invalid or not specified Unknown

 

[YiannisH_3CX]

I believe you misunderstood what i was trying to say. Let me try to explain a bit better. The issue you are describing has 2 parts. The first part is that services failed for some reason. The other part is that services failed to start automatically when you restarted the machine.

The first part of the issue could have been caused by the machines resources or a reboot of the machine of a power failure etc. That is why i suggested making sure that the machine was not rebooted at that time of there was a power failure etc.

The second part of the issue is in most cases caused because the 3CX services try to start before the network service starts. That could be seconds apart or even less but because there is no network services available when the PBX services try to start the services fail to start. I never suggested that services did not start because you lost internet or network connectivity.
Some 3CX services require a network to bind when they start so if the network interface is not up then this happens services will fail.

Having said that i would recommend upgrading the machine to Debian 9 as Debian 8 is not supported with the latest service pack releases and see if the issue persists.