Solved Bizzare behavior today - requesting troubleshooting ideas

[MichiGanger]

We haven't made any changes to our system and things have been working great for months. Today around 1pm we noticed that callers were clear for us but they were receiving garbled audio. Inbound and outbound. Also, we just realized that when they leave voicemails we arent getting email notifications and when we log into the voicemail extension, they are not server-side either. I restarted my 3CX services and also restarted the google cloud instance. The CPU was at 100% for quite some time. Other things we noticed - our weekly backups stopped on 12/15 and we havent had any new ones. I tried to open the event log and got an nginx error and then it logged me out of the management console. I am unable to log in now, but our phones seem to be working and voicemails are coming through.

If anyone can help me with this I would greatly appreciate some advice. If there is some way to retrieve all of the voicemails from 1-5pm that would be fantastic. Also, I'm not sure how to get back into the management console, and I'm not sure if I should restart the google cloud instance again.

thanks in advance!

 

[cobaltit]

I'd start with this:

https://www.3cx.com/community/threads/information-to-provide-when-requesting-help.67558/

 

[MichiGanger]

Sorry - that would be helpful wouldn't it!

3CX Version: Professional Annual 16.0.493
Server: debian-9-stretch-v20180206; 10gb; 1.7GB memory
Is the 3CX Server Hosted and where? Google Cloud
IP Phone Make/Model/Firmware: Fanvil X4/X2 2.10.1.6836;
Provisioning Method: SBC (3 locations, 2 windows 16.0.390, one raspbian 16.0.390)
Trunk Provider: Flowroute
Has the Firewall Checker passed: YES
Are custom Phone Templates being used: NO

 

[leejor]

Were internal calls affected at all? Do you just have the one provider? Have you spoken to them to see if they were having any issues?

 

[MichiGanger]

so, i realized that the VM disk was 99% full due to thousands of voicemails. deleted them and increased the disk size.

 

[leejor]

That's what happens when people save their voice-mails "forever", just in case...

 

[cobaltit]

An easy enough fix. I'd recommend using the Voicemail Quota settings in 3CX to prevent this from happening in the future.