Solved 3CX v16 disk space usage

[Tahir]

We have 3CX installed at OVH and its working great and has been for some time, i believe the VM has 10 GB disk space. The 3CX dashboard is showing 87% used, 1.2 GB free, I cannot understand where the storage is being used though as they have no call recording saved, not sure if voicemails are seperate but there is nothing in there either, backups we have only 100MB worth, 4 copies.

Where is the rest of the storage being used/consumed and is there a way to optimise this or should I increase the disk space to 20 GB instead?

I am just uncertain if 3CX is clearing up any downloads for upgrades/firmware etc after it has been installed.

Can anyone possibly assist?

 

[BrenttG]

10 GB for any VM is very small, better to always default to at least 20 or 25 to be safe. We default to 50 so there is always recording space and extra.

 

[Lee Cramman]

Whenever disk space disappears on an established server the first thing I look at are log files.

That said, 10Gb is pretty paltry... 20Gb is probably a sensible minimum.

 

[AWS2P]

Do you use Chat? if so try to purge

 

[Tahir]

We dont use chat, and I have purged all records and items I can from the dashboard but the usage still appears to be going up rather than down.

Where does 3CX keep the upgrade files and firmware updates? Once these have been installed are they auto-deleted or kept on the box as this could be the reason behind the sudden increase of storage.

 

[Tahir]

I would really appreciate a response from 3CX as when I first raised this post i had 1.2GB free, i now have 300 MB free and nothing has changed.

 

[Tahir]

So, the saga continues. I have had to upgrade the disk on OVH to 20 GB as i only had 200 MB free which was also disappearing fast. I have tried to login to SSH to look at the system with my limited knowledge but cant login as dont have a SSH key which was supposed to be generated by the 3CX Wizard installer which was used to deploy the instance and without it, its not possible to get direct access to the system. I seem very stuck and a little annoyed about the 3CX Wizard not providing these details as everything that was provided is written down.

Any ideas how to get a SSH key pair now to login to OVH when 3CX failed to provide one?

 

[AWS2P]

you normally have this in your OVH Manager no?

 

[Lee Cramman]

Are you sure you get this via the wizard with OVH? You don't when you deploy via Google... you get it from the VPS providers portal.

3CX won't be able to help you with an OVH problem because they do not hold your SSH key.

Once you have gained SSH access you can try this to find out what is taking up all of your space, however be very careful with step 5!

https://maryrosecook.com/blog/post/freeing-disk-space-on-your-linux-server

I'm not sure a 3CX response is likely to say more than the hardware requirements for 3CX state 30Gb disk size as an absolute minimum (often more than that, depending on your number of extensions) so you should start by expanding space to 30Gb.

Finally, if you haven't been able to SSH in then expanding the size of the disk alone may not increase your free space (I'm not sure how it works with OVH but this is certainly the case with Google). You may also need to expand the partition to fill that free space you have created which is usualy done in an SSH session. I would suggest you check your free disk space again just to be sure you have actually fixed the problem.

 

[Tahir]

Thanks Lee. Yes it seems on the PBX Express guide it says this:
Go to “Compute” > “Key Pairs” tab and click on “Create Key Pair”. Enter a name for this key (this Key will be visible later in the PBX Express wizard). If you want to connect to the machine using SSH, press “Copy Private Key to Clipboard” and store this in a text file.

But in fact there is no option to copy Private Key, it seems it downloads it which I missed, and at the time I must have skipped past this very important step. It would have been advisable to note that its possible to change from needing a SSH key to a password after install as another step should one wish to revert back to passwords but that was also not mentioned in the guide.

I have spoken with OVH and there is a way to add a new key but it seems the recovery mode on the VPS is not working correctly and their administrators are taking a lifetime to respond. Basically when the VPS is put into recovery mode it generates a temporary username and password to allow you to SSH and add the new key or remove it entirely and set it back to a standard SSH password but its not recognising the temporary password so I cant do anything.

I have time at present as I have purchased the extra storage but something is continuing to eat around 100 MB every hour or so in the background on the OS. And purchasing the extra storage has actually worked as if it had not the whole 3CX system would currently be offline due to 0% free space. I am glad it works better than Google at OVH, life is much better without the complexities especially when you are not very good at Linux/Unix.

What does the command ls- l do exactly? Does it provider a directory listing or is it a listing which if you press the wrong thing will also delete the files?

 

[Zol]

What does the command ls- l do exactly? Does it provider a directory listing or is it a listing which if you press the wrong thing will also delete the files?

"ls -l" just list the files in a given directory. To delete a file use the command "rm filename".

 

[Tahir]

Thanks just thought I would double check as you said to be very careful with the command

 

[AWS2P]

But in fact there is no option to copy Private Key,

What browser do you use ? i've done only one time on OVH with PBxpress, but i remember having ability to save the key

 

[Tahir]

I use Google Chrome primarily. When i tried it today generating a new key pair it downloaded the .pem file automatically but at the time i was not aware of using .pem SSH key files for this purpose so had copied the OVH Stack Username and Password as instructed in the guide rather than looking for a .pem file which contained the private key. The guide document should have been a lot clearer on this aspect given the importance of the details.

It would also have been super nice if the terminal within 3CX allowed more access to maybe delete files consuming space or browse the directory tree so problem solving could be conducted without the need to directly SSH. Hell, even full terminal access would be nice and i think the OVH engineers were hoping it had that but I had to explain its got limited commands that are usable which are really of little use given the circumstances.

 

[AWS2P]

on ovh manager ssh key is always available

 

[Tahir]

That is the public key, not the private key. - I apparently need the private key which was not saved at the time of setup/creation which needs to be linked into Putty to allow me to SSH.

 

[Lee Cramman]

I'm not an OVH user myself but surely they allow you to create a new private key from the dashboard or provide some other method of gaining terminal access?

I suppose your other option would be to completely blow away the VPS and start again using your backup, assuming you can afford the downtime.

 

[Tahir]

I'm not an OVH user myself but surely they allow you to create a new private key from the dashboard or provide some other method of gaining terminal access?

I suppose your other option would be to completely blow away the VPS and start again using your backup, assuming you can afford the downtime.

Yes OVH have said I can generate a new key and add it and 3CX have confirmed via support that changing the SSH key should not cause any issues with the installation. It seems OVH had an issue where when the VPS was booted into rescue mode where a temp username and password are provided so I can SSH to fix the issue the credentials provided were not being accepted so the ticket had to be escalated. They confirm this should work now and will assist me around 5pm today so will soon find out.

In the meantime whatever is consuming space on the box continues to do so, its driving me nuts though not being able to SSH in and try to figure out the cause of the issue when I know there is a time-limit in place before all space is consumed and the system will fail to boot.

 

[Tahir]

Just to provide an update. I spoke with OVH tech support who assisted very briefly to change the SSH though we opted for a password instead of a SSH key in this instance. This allowed me to connect via SSH onto the VPS and locate the cause of the space issue.

Information on how to do this is very basically outlined below via the OVH tech as reference for anyone else or myself in the future:

'Regarding adding a password on your server, you will need to enter the following command line after mounting your disks and chroot in the disk: passwd

You will need to change your SSH configuration on your system to allow password authentication. It is usually stored in /etc/ssh/sshd_config. It will be best asking 3CX before to make sure it won't interfere with their software'

It appeared the space was consumed by logs, the instructions provided were:

'When you gain SSH access to the server, run the following command to see the largest directories on the system:
sudo du -Sh | sudo sort -rh | sudo head -5
For this to work you need to be in the root directory (not /root, the one above it)

Based on the result of the commands, the space is consumed by the backup logs. At least most of it is (about 7GB).

Through the management console, please navigate to Dashboard>>Activity Log>>Settings and lower the days that the PBX will keep backup of the logs.
Also, please clarify if the PBX is in verbose logging level (visible in the same window and the above option).

If you need to have these backup logs stored, you need to consider increasing disk space.'

I was advised its possible to delete the logs using the command below:

You can delete the backup logs folders to clear space on the system (these are located under /var/lib/3cxpbx/Instance1/Data/Logs/Backup)
The logging level does not change with updates.