Phones behind SBC displayed 'No service', resolved by PBX reboot. What's happened?

[Leo_B]

Well, this Friday I woke up because of the customer's phone call. It was reported that office phone system has stopped working. I checked the one, no alarms in the Dashboard, all trunks are okay, SBC up and running, yet office phone show 'No service' status. Resolved the problem by restarting the cloud VM running PBX, yet still need assistance in inbestigation what exactly happened. Could someone share the hints where should I look and what should I check to get more details on Friday's incident described?

Thank you in advance!

3CX Version: 3CX Enterprise 8 SC 18.0 Update 8 (Built 912)
Server OS: Debian 10 4.19.282-1
Is the 3CX Server Hosted: no (Vultr-powered VM)
IP Phone Make/Model/Firmware: Yealink T58A
Provisioning Method: SBC
Trunk Provider: Vitelity
Has the Firewall Checker passed: YES
Are custom Phone Templates being used: NO

 

[DialogueVoice_Zack]

Good Morning,

Although I personally never had this happen to any of the sites I manage, I may be able to offer some assistance with troubleshooting.

What type of hardware do you have the SBC running on?

 

[Leo_B]

What type of hardware do you have the SBC running on?

Pi 4 running Linux raspberrypi 5.4.51-v7l

 

[DialogueVoice_Zack]

Hello,

I would recommend starting by checking to see if you ran out of memory. The following 4 commands will check various logs to see if there were any out-of-memory conditions:

journalctl | grep -i "out of memory"
dmesg | grep -i "out of memory"
cat /var/log/syslog | grep -i "out of memory"
cat /var/log/messages | grep -i "out of memory"

Also, let's check the swappiness setting:

cat /proc/sys/vm/swappiness

Finally, it would be good to check if there are any power\voltage issues. I had an issue a few months ago where a power supply for an ATA and a power supply for a Raspberry Pi got switched during an installation. The ATA power supply was 5V 1.5A and the Raspberry Pi power supply was 5V 2.5A. The ATA was fine but the Pi was undervolted and not performing correctly.

Check if there are any voltage messages in the logs (which may\should show voltage issues historically):

dmesg | grep -i "voltage"

And check if there is currently a voltage issue:

vcgencmd get_throttled

If the output is 0x0 or 0x00000000 there are no power throttling. If it is anything else, such as 0x5 or 0x5005 I would recommend changing the power supply.

Finally, on some Linux boxes I have (both SBCs and other small servers) I set up a cron to reboot the box once a week, day, or month.

First, if you are not familiar with the vim editor, change your primary editor to nano:

echo "export EDITOR=nano" >> ~/.bashrc
source ~/.bashrc

Open up the system crom:

crontab -e

And paste in one of the following depending on how frequently you want the cron to run:

# once per month, on the 1st of the month at 3AM
0 3 1 * * reboot

# once per week, on sunday at 3AM
0 3 * * 0 reboot

#once per day, at 3AM
0 3 * * * reboot

Let me know the output of the above and advise if I can do anything else to help with your troubleshooting!