Toggle sidebar Toggle sidebar

Solved Linux VM freezing since upgrade to v18

Status
Not open for further replies.

mackey

Free User
Basic Certified
Joined
Jan 31, 2022
Messages
18
Reaction score
7
Hello All,

I have a 3cx instance running on ESXi installed using the standard ISO image from 3CX.
Since upgrading to version 18 the whole VM is freezing/locking every 2-3days. I have increased the RAM and vCPU but the problem continues.
I have even rebuilt the VM from scratch and restored the a backup config.

Anyone else having this issue?
 
mackey said:
Just to add, my 3CX crashed again today. :(
Click to expand...
3CX Crashed, or the whole VM that 3CX is installed on became unresponsive?
Especially if it's the 2nd, I'd read the comment right above yours from @KVCIT .
 
The whole VM freezes on my ESXi host, even the console screen is frozen and a full restart of the 3CX guest is required.

I've seen the comments regarding the network connectivity. I'll see if I can replicate it.
 
sburgess73 said:
Get off your VM and run it on a Raspberry PI device. Easier and less maintenance.
Click to expand...
Yeah, because that will go down well with a client. "Hey bud let's move your customer facing PBX from fully redundant server stack to a 9VDC circuit board in a plastic fisher price case :)
 
small comments in case helpful re: this crashing. If I have read the thread correctly, you were able to temporarily improve things by doing console root commands, "apt-get update" and "apt-get dist-upgrade". It might be good to re-try those and see how they fare. I've seen a few cases on debian (not 3cx specific, just debian) where things had been configured to auto-update. But these auto-updates fail in the case of a 'release version increment' that requires human input/intervention for the version change to be 'approved' (ie, move of stable deb from 11.2 to 11.3 and approve that this change is ok). It does not make sense to me really that this could cause a lock up in your case though. But an easy 'related' thing possibly.

That being said for what it is worth. I've got a bunch (+>12) of client VM instances of 3cx, 3cx-SBC (basically same thing but more stripped down install, I believe?) - running on any of

Proxmox/KVM
OpenStack (OVH public cloud - KVM based I believe)
a few on Synology NAS (KVM custom environment, kind of)
and of course some bare metal (Lenovo tiny typically)
they were all installed in same manner (stock 3cx ISO debian)
(exception, I guess the OVH installs uses the "PBX Express install" which is mediated via 3cx > OVH but is effectively under the hood not too dissimilar I think).

and none of them appear to be having lock up behaviour with any/current/latest 3cx version.
other things I can think to suggest to check?

- make sure your vmware underlying datastore is healthy, has free space? I know random VM Lockups sometimes get hidden cause 'hiding in plain sight' if you had one VM with thin-provision disk space (ie, 3cx) and others with thick-provision / and then underlying data store in vmware fills / and all your thick-disk VMs are happy/ok but thin-provision ones are sad/crash/freeze.

- in theory the open source vs vmware provided vmware tools should not be a factor.. But that being said I don't have any 3cx running on VMware so have no specific comments to add of value here, other than - debian does normally run fine on vmware / it is a pretty standard and well supported environment. and 3cx is (more or less) using stock debian ISO / stock repo install / with a bit of their own customization for the 3cx PBX or SBC install/features.

sorry if these comments are entirely not helpful. but figured I would mention a few thoughts in case they prove to be of any./maybe/slight use.

Tim
 
  • Like
Reactions: KVCIT
sburgess73 said:
Well what you have at the moment doesn't seem to be working. 3CX would not cause a VM to lock up like you are saying. Sounds like a host issue.
Click to expand...
It's not a host issue, this problem occurred when we upgraded to version 18 of 3CX. 3CX was rock solid on the previous version and is rock solid with our other production servers, on the same host. Until we can get to the bottom of the issue on our own system we can't possibly risk upgrading any of our clients.
 
fortech said:
small comments in case helpful re: this crashing. If I have read the thread correctly, you were able to temporarily improve things by doing console root commands, "apt-get update" and "apt-get dist-upgrade". It might be good to re-try those and see how they fare. I've seen a few cases on debian (not 3cx specific, just debian) where things had been configured to auto-update. But these auto-updates fail in the case of a 'release version increment' that requires human input/intervention for the version change to be 'approved' (ie, move of stable deb from 11.2 to 11.3 and approve that this change is ok). It does not make sense to me really that this could cause a lock up in your case though. But an easy 'related' thing possibly.

That being said for what it is worth. I've got a bunch (+>12) of client VM instances of 3cx, 3cx-SBC (basically same thing but more stripped down install, I believe?) - running on any of

Proxmox/KVM
OpenStack (OVH public cloud - KVM based I believe)
a few on Synology NAS (KVM custom environment, kind of)
and of course some bare metal (Lenovo tiny typically)
they were all installed in same manner (stock 3cx ISO debian)
(exception, I guess the OVH installs uses the "PBX Express install" which is mediated via 3cx > OVH but is effectively under the hood not too dissimilar I think).

and none of them appear to be having lock up behaviour with any/current/latest 3cx version.
other things I can think to suggest to check?

- make sure your vmware underlying datastore is healthy, has free space? I know random VM Lockups sometimes get hidden cause 'hiding in plain sight' if you had one VM with thin-provision disk space (ie, 3cx) and others with thick-provision / and then underlying data store in vmware fills / and all your thick-disk VMs are happy/ok but thin-provision ones are sad/crash/freeze.

- in theory the open source vs vmware provided vmware tools should not be a factor.. But that being said I don't have any 3cx running on VMware so have no specific comments to add of value here, other than - debian does normally run fine on vmware / it is a pretty standard and well supported environment. and 3cx is (more or less) using stock debian ISO / stock repo install / with a bit of their own customization for the 3cx PBX or SBC install/features.

sorry if these comments are entirely not helpful. but figured I would mention a few thoughts in case they prove to be of any./maybe/slight use.

Tim
Click to expand...
Thanks for taking to the time to reply. After the most recent lock up today I have reran the distro update command and it pulled in a further update for 'libc-bin' its now on v2.28-10. Hopefully that may help.

If it doesn't, your suggestion of migrating to a think provisioned VMDK is my next step.
 
I had a similar crash issues - in my case was related to Scheduled backups and free space on HDD.
With time the size of backup grown to the substantial amount. For the backup to take place it first compresses it and then sends to external destination - my disk space was not enough to hold the backup file before the export thus resulting in the crash. Disabled backups for few days to confirm and no more crashes.
 
darn - forgot to test over the weekend! -
I'll try to test mine this weekend..

what is your firewall @mackey?
(also SIP Trunk / Using PRI > SIP?)

Ours upgraded to v18 early on..
so its odd that it has frozen only on network issues --

All other VMs (similar VM is unifi controller - keeps chugging along and when network is restored all good - doesn't freeze up)

Very similar Freeze... console shows login - but is frozen.. reboot resolves
SSH / Ping all un-responsive
 
KVCIT said:
darn - forgot to test over the weekend! -
I'll try to test mine this weekend..

what is your firewall @mackey?
(also SIP Trunk / Using PRI > SIP?)

Ours upgraded to v18 early on..
so its odd that it has frozen only on network issues --

All other VMs (similar VM is unifi controller - keeps chugging along and when network is restored all good - doesn't freeze up)

Very similar Freeze... console shows login - but is frozen.. reboot resolves
SSH / Ping all un-responsive
Click to expand...
We have a single SIP trunk and the firewall its sat behind is a PFSense virtual machine (one the same host).
 
@mackey

May I ask exactly which VMware ESXi version you are running this VM on?
 
ChrisC_3CX said:
@mackey

May I ask exactly which VMware ESXi version you are running this VM on?
Click to expand...
Sure, it's: HPE Customized Image ESXi 6.5.0 version 650.10.1.5 released on October 2017 and based on ESXi 6.5.0 Vmkernel Release Build 5310538
 
  • Like
Reactions: KVCIT
  • Like
Reactions: KVCIT
mackey said:
Thanks for the tip Chris. Indeed my 3CX VM is using the VMXNET3 vNIC, if the lockup occurs again, I try swapping it for the standard E1000 vNIC
Click to expand...
I would actually think it's best to upgrade VMware ESXi's version.
 
Did you say what version of esxi you were using. We had a problem a few years back related to exsi 5.? This also was related to using the VMXNet instead of legacy network adaptor. Was a know issue with a hard lock up as you describe.
Pretty sure there was an Update or we moved to 6.? at that time and resolved the issue. A the time changing NIC also worked.
 
3CX is tested and supported to run as a Virtual Machine on these hypervisor platforms:

  • VMware vSphere Hypervisor (ESXi) 6.5u1 and

What is your version? We needed to upgrade vmware before moving from v16 to v18, we got older version of vmware.
 
ChrisC_3CX said:
The reason I asked is because there's actually a known VM freeze issue with ESXi 6.5 and Debian 9. You are running Debian 10 of course but it might be worth considering upgrading the hypervisor version too just in case.
Click to expand...
Very nice - I'm checking this as well..

Indeed 3cx is setup with VMNet3 vs E1000 -

Unifi VM is E1000 same OS but makes sense now that 3CX Deb Host was only one affected!

6.5u1 - so its a bit behind but the client doesn't pay for a lot of maintenance

Will switch once the office closes today to E1000 and probably just start updating to 6.7 anyway to keep some hair! or at least U3
https://www.virten.net/vmware/esxi-release-build-number-history/#esxi6.5

-----

Virtual PFSense - Like it..
I have some behind Virtual Fortigates and others no issue.. but most are ESXi 6.7>7.0

and yes even a small client on a Raspberry Pi - 3CX Kit - looks much better than plastic cases
https://www.canakit.com/3cx-raspberry-pi-kit-pbx.html

updating ESXi - comes with its own issues at times - had one client no long could access data store due to 7.0 not having the correct driver for an older Dell Hypervisor... rolled back to 6.7 fine but naturally had updated new server and vms to v7 hardwardware (v11 or something) so can't move them to 6.7 due to that!.. grrrr..
 
  • Like
Reactions: ChrisC_3CX
Hello All,

quick update, since we switched the virtual NIC to an E1000 the 3CX has been solid.
 
mackey said:
quick update, since we switched the virtual NIC to an E1000 the 3CX has been solid.
Click to expand...
Excellent, hopefully that indeed was it!

As previously mentioned, I do also recommend upgrading VMware if you can, but that's of course up to you.

I'll be marking this thread as solved for now so please let me know if anything changes.
 
Status
Not open for further replies.

Latest Posts

Forum statistics

Threads
111,974
Messages
590,081
Members
164,899
Latest member
mazet
3CX Logo Footer Copyright © Sitemap Privacy
Top Bottom