Backup Issues - V20 U7 (Multiple Systems)

[Drain Bamaged]

W2019 Server - Version 20.0 Update 7 (Build 1080 Release) - OnPrem

As of today we have 6 out of 9 Enterprise licensed systems that are no longer backing up. We used to do local backups, then DFS'd the ZIP files to other systems, but with the mandatory Remote Storage changes we began using SFTP. Everything worked for a few months and now things are just breaking. No system / network or SFTP site changes on our side and these servers are in different countries for the most part.

They stopped working Nov - Dec and we cannot get the automated backups to run. There is nothing in event log during the times backups go off. I deleted the .tmp thinking it would fix it, but nothing happened.



Manual backups work, automated do not. I changed the backup name on the file to something different, restarted services and rebooted the systems to no avail. The only other thing I noticed was a .tmp file that stopped after 10% or so. Deleting it, did nothing.

I'm stumped.

Edit: Just before I posted this I changed the SFTP target to SMB and set up a file server share for a test to see if it was the SFTP site. I still get the .tmp file on auto backup and it stops there. Manual backups work:

 

[Charles_3CX]

Regarding your question, and mentioend issue, this behaviour points to a permissions issue affecting scheduled backups, not SFTP or networking.

Scheduled backups first create a *.zip.tmp file and then rename it to the final ZIP. If the backup user does not have modify/rename permissions, the process stops at the .tmp stage. This is why manual backups work, but automated ones fail.

Please check:

• The backup user (e.g. 3cxbackup) has Modify / Write / Delete rights on the backup folder (share and NTFS if SMB).
• There is no existing 3cxScheduledBackup.zip.tmp file in the destination.
• Permissions allow file rename operations.

This applies to both SMB and SFTP and matches the symptoms you’re seeing.

 

[Alphabetic]

Also check for any anti-malware security you have as your AV could be seeing this as ransomware / encryption.

 

[Drain Bamaged]

Regarding your question, and mentioend issue, this behaviour points to a permissions issue affecting scheduled backups, not SFTP or networking.

Scheduled backups first create a *.zip.tmp file and then rename it to the final ZIP. If the backup user does not have modify/rename permissions, the process stops at the .tmp stage. This is why manual backups work, but automated ones fail.

Please check:

• The backup user (e.g. 3cxbackup) has Modify / Write / Delete rights on the backup folder (share and NTFS if SMB).
• There is no existing 3cxScheduledBackup.zip.tmp file in the destination.
• Permissions allow file rename operations.

This applies to both SMB and SFTP and matches the symptoms you’re seeing.

Hi Charles,

The SMB user has full control over the file share.
The SFTP user has full control over the SFTP folder.

Manual backups work fine using the same account, it is the auto-backups that fail.

The .tmp file is able to be created, it gets about 10% into the backup and hangs.

 

[Drain Bamaged]

Also check for any anti-malware security you have as your AV could be seeing this as ransomware / encryption.

All of our AV is consistent in all jurisdictions. If this were the case ALL backups would be affected given the target destinations are the same, just different sub-folders. We have repositories in EU and NA and we're seeing inconsistent backups in both locations. Manual backups work fine to the same folders.

There are no reports from AV about blocked write attempts.

 

[Charles_3CX]

Try checking the share permissions on the backup location itself,

For example:



Then check the security in the event log

 

[Alphabetic]

Your 3CX is on prem with your server? Why are you not using SMB share?

Have you tried checking the SMB logs on the server?

 

[KyriacosS_3CX]

Hi @Drain Bamaged ,

We recommend that you open a ticket on this issue, so the team can take a proper detailed look. It will be very difficult for the community to home-in on what's happening without looking deeper into logs and so on.