Scheduled backups fail to launch, but manual backups succeed

[Nick Kennett-Brown]

We are managing 2 separate 3CX environments, one for a customer and one for ourselves.

Both systems are set to run scheduled backups to an SMB share (separate folders for each system). The first system stopped doing scheduled backups on the 2nd December. When it stopped I checked the SMB storage and found the .tmp file left (it was well after the backup would have finished). I removed the .tmp file and expected the scheduled backups to resume (this had resolved it in the past). This time the system has failed, as far as I can tell in the logs, failed to even attempt the scheduled backups.

The second system stopped scheduled backups on the 24th December, again the .tmp file was present and removed, but now in the same state as the first.

Both systems are correctly storing recordings to the SMB share and manual backups run and complete correctly. Manual backups show in the logs as well, so it is a mystery to me why there is nothing for the scheduled backup, no start message, no failure messages, just nothing around the time it should launch.

So far I have tried the following:

1) Changing the scheduled backup time and saving
2) Changing the SMB path to a sub folder
3) Removing prior scheduled backups to a subfolder
4) Restoring the prior scheduled backups to the normal location
5) Disabling Remote Storage, save, enable Remote Storage, re-entering the account password
6) Checked the account permissions on the SMB location to make sure it has Modify permissions

I haven't manually rebooted either system, but as far as I am aware they both reboot weekly. I have verbose logs from one system from last night (the scheduled backup was set to run at 20:01) but it only captured the manual backup I ran at ~09:00 this morning. I can provide this zip as needed. Both systems are running v20.0.7.1080

Any suggestions gratefully received

 

[Nick Kennett-Brown]

As an update, in case others run into the same issue... I rebooted our 3CX system from Admin > Dashboard > Restart OS and that cleared the issue. It is strange, as far as I know, it is set to automatically reboot each week, but it cleared the issue for our 3CX.

I have scheduled a 3CX reboot with the customer for tonight and so I'll reply here with the results tomorrow.

 

[Nick Kennett-Brown]

So the customer's 3CX was rebooted and then the scheduled backup ran correctly overnight, so I guess the old IT adage of "Have you tried turning it off and on again?" kicked in again.

Thought I'd update the thread in case someone runs into this again in the future.

All the best for the new year to everyone

 

[ivank]

Looks like a stuck backup job, they are not allowed to run several instances concurrently. So if this happens next time, please collect logs and provide feedback so we could figure out what caused this.
Also if I am right, you don't need to restart the OS entirely - just 3CXSystemService.

 

[ivank]

We are managing 2 separate 3CX environments, one for a customer and one for ourselves.

I would keep your instance in Verbose if possbile so we can track back when and where the issue started.

 

[Nick Kennett-Brown]

I have the verbose logs from the day before yesterday, but the normal logging settings do not keep them on verbose logging due to space restrictions. The issue started on the 2nd of December and the logs don't seem to go that far back. At least as far as I can tell.

What I could tell was that the scheduled backup did not even try to start, the only entries I could find for Backup or similar was the manual backup I started to keep the customer protected.

I can send you a copy of the logs if you'd like, if you let me know where to upload them to, would prefer not to the open forums if possible due to customer data.

 

[ivank]

I need verbose logs next day after the last successful backup (i.e. first failed one), that's why I suggested to keep verbose and wait until next time.

 

[Nick Kennett-Brown]

I'll see what I can do but as far as I am aware this is the first time we've ever seen this issue for either system and they have been running for long than I have been with the company (I started in August).

Normally when we see a backup fail it is due to the .tmp file of a previous backup still being present in the remote storage and removing it resolves the issue.

 

[ivank]

Could you check your email and PBX event log for any backup errors back then?