Backup Failure - Maybe related to size OR daylight saving clock change

Status
Not open for further replies.

Alan9846

Silver Partner
Joined
Aug 24, 2019
Messages
96
Reaction score
19
Hi All,

I am getting a backup failure on one of my 3CX servers. It is running 3CX v16 on Debian 9.

Backups were working fine up to and including Sunday (6 Oct 2019) morning at 3am, but began failing as of Monday (7 Oct 2019) at 3am.

If I try to run a manual backup, I get the 'failure' email report too, so it is not just the scheduled 3am backups.

The timing is suspicious in that the clocks here changed forward one hour on Sunday morning as part of the normal spring change to summer-time, but both 3CX and the Debian host report the correct date and time (if I were to ask the proverbial 'man on the street' what time it is right now, they would agree with the Debian server).

I also have another server with the same setup (different business), and that one has carried on without any issue. It is possible there is a difference between the two Debian setups, but I would be mildly surprised if that was the case - nobody but me has accessed either server.

Settings are to backup everything, including voicemails and recordings, but not to use a password. In actual fact, no voicemails are stored on the server - they all get emailed to the user.

Backups are run to the local disk (I copy them over to an offsite store via SSH weekly). I have checked the disk-space, and there is more than 200GB on the drive / partition that the backups go to. The backups, up to Sunday morning, were all about 200MB each (I checked back a few weeks - they are all essentially the same size), so it does not appear that size or space is the problem and I should be able to store about a thousand more such backup files, however....

If I turn off everything but the 'License Key Information, FQDN & Conference' option (which reduces the backup file size to about 10MB) the backup runs fine, and I get a 'success' email report, which indicates a space issue despite the fact there is plenty available.


For now, I am running the backups manually with the only the 'License Key Information, FQDN & Conference' option selected, but I need to resolve this.

I have included below, the full error message I am getting in the email. All the error reports are the same (except for the date and time). I have replaced some info with XXFieldDescriptionXX, but I don't believe it hides anything important.

Thanks for your suggestions / advice,

Alan.


Code:
3CX Phone System - XXClientXX - XXSuburbXX - XXCityXX <[email protected]>

Oct 7, 2019, 3:00 AM (3 days ago)

to XXReportingEmailAddressXX

3CX Phone System tried to backup but failed due to the following reason(s):
System.Exception: System.IO.IOException: Input/output error
   at System.IO.FileStream.CheckFileCall(Int64 result, Boolean ignoreNotSupported)
   at System.IO.FileStream.ReadNative(Span`1 buffer)
   at System.IO.FileStream.ReadSpan(Span`1 destination)
   at System.IO.FileStream.Read(Byte[] array, Int32 offset, Int32 count)
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
   at _3cxNetBackupAndRestore._3cxBackup.ZipFile(String dst, IFile src)
   at _3cxNetBackupAndRestore.BackupTasks.BaseFilesTask.BackupFiles()
   at _3cxBackupRestoreImpl.BackupRestoreUtils.Backup(IFilesystem filesystem, String instanceId, String cfgPath, String dbConnectionString, BackupConfig options, String fullpathFilename, Boolean sendEmail)
   at BackupRestoreSchedule.BackupJob.Execute(IJobExecutionContext context)
 
Last edited:
Hi Alan,

Firstly you should return the Backup folder to the 3CX default and check again. Does it work there?
/var/lib/3cxpbx/Instance1/Data/Backups

If that is ok for a full backup, then check the permissions in your custom destination folder: does the user phonesystem have rights to write? Are you certain that the custom folder has space or are you just looking at the management console disk usage graph and assuming that you have free space?
12621
 
Hi JohnS_3CX,

The current backup location is:

/var/lib/3cxpbx/Instance1/Data/Backups

What makes you think that it is different? Is there something in the failure report file that gives you that impression?

When I run a backup with the reduced options, it goes into that folder fine, but when I run with all the options (except password) it fails - both to the same folder.

If it was a permissions issue, then wouldn't the reduced scope backup also fail?

This is a Debian install, so disk space usage and availability were determined by the 'df' command, and in the case of the backup sizes, using 'ls -la /var/lib/3cxpbx/Instance1/Data/Backups/'

For the avoidance of doubt, whilst I omitted it above, the first thing I did was reboot the server, but that makes no difference.



Thanks,

Alan.
 
Last edited:
Just for humor can you paste the output of the following command
df -h

Then
cd /
du -sh *
 
Hi BrenttG,

This is what I get. My reading is that I have 208GB of space on the partition to play with, and only 6.2GB in the whole of /var/.

Code:
root@3cx:~# df -h

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           785M  8.9M  776M   2% /run
/dev/nvme0n1p1  226G  7.5G  208G   4% /
tmpfs           3.9G   16K  3.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           785M     0  785M   0% /run/user/0


root@3cx:~# cd /

root@3cx:/# du -sh *

9.4M    bin
64M    boot
16K    dev
4.3M    etc
24K    home
0    initrd.img
0    initrd.img.old
490M    lib
4.0K    lib64
16K    lost+found
20K    media
4.0K    mnt
4.0K    opt
du: cannot access 'proc/29888/task/29888/fd/4': No such file or directory
du: cannot access 'proc/29888/task/29888/fdinfo/4': No such file or directory
du: cannot access 'proc/29888/fd/4': No such file or directory
du: cannot access 'proc/29888/fdinfo/4': No such file or directory
0    proc
32K    root
8.9M    run
6.7M    sbin
4.0K    srv
0    sys
8.0K    temp
32K    tmp
678M    usr
6.2G    var
0    vmlinuz
0    vmlinuz.old

root@3cx:/#


Do you agree?

Thanks,

Alan.
 
Hi Alan,

Sorry, when you only said "I have checked the disk-space, and there is more than 200GB on the drive / partition that the backups go to" so I may have been thrown off thinking you selected another destination, no biggie :)

Is there any chance your system is not the owner for some folders in the 3CX installation main folder?
First thing I check is for user phonesystem rights for these kinds of problems, and reduced options means it will backup less stuff from various folders (ie. skip firmware, voicemail or recording folders) hence not throwing an error in that case
 
  • Like
Reactions: BrenttG
Hi John,

No worries - I see how it could have meant I was doing that.

In terms of ownership / permissions, everything appears to be owned by 'phonesystem':

Code:
root@3cx:/var/lib# ls -la

total 92
drwxr-xr-x 23 root        root        4096 Aug  8 10:48 .
drwxr-xr-x 12 root        root        4096 Jul 24 17:36 ..
drwxr-xr-x  6 phonesystem phonesystem 4096 Oct  6 04:00 3cxpbx
drwxr-xr-x  3 root        root        4096 Aug  8 10:48 apparmor
drwxr-xr-x  5 root        root        4096 Oct  6 04:01 apt
drwxr-xr-x  2 root        root        4096 Jul 24 17:33 dbus
drwxr-xr-x  2 root        root        4096 Jul 24 17:33 dhcp
drwxr-xr-x  7 root        root        4096 Oct  6 04:01 dpkg
drwxr-xr-x  2 root        root        4096 Aug  8 10:48 initramfs-tools
drwxr-xr-x  2 root        root        4096 Oct 12 06:25 logrotate
drwxr-xr-x  2 root        root        4096 Mar 28  2019 misc
drwxr-xr-x  7 root        root        4096 Jul 24 17:36 nginx
drwxr-xr-x  2 root        root        4096 Jul 13  2017 os-prober
drwxr-xr-x  2 root        root        4096 Jul 24 17:33 pam
drwxr-xr-x  3 postgres    postgres    4096 Jul 24 17:36 postgresql
drwxr-xr-x  2 root        root        4096 Jul 24 17:36 python
drwxr-xr-x  2 root        root        4096 Jul 24 17:36 sgml-base
drwx--x--x  4 root        root        4096 Jul 24 17:44 sudo
drwxr-xr-x  7 root        root        4096 Aug  8 10:49 systemd
drwxr-xr-x  3 root        root        4096 Jul 24 17:36 ucf
drwxr-xr-x  2 root        root        4096 Jul 24 17:31 usbutils
drwxr-xr-x  3 root        root        4096 Jul 24 17:31 vim
drwxr-xr-x  2 root        root        4096 Jul 24 17:33 xml-core

root@3cx:/var/lib# cd 3cxpbx/
root@3cx:/var/lib/3cxpbx# ls -la

total 28
drwxr-xr-x  6 phonesystem phonesystem 4096 Oct  6 04:00 .
drwxr-xr-x 23 root        root        4096 Aug  8 10:48 ..
drwxr-xr-x  3 phonesystem phonesystem 4096 Jul 24 17:48 .aspnet
drwxr-xr-x  5 phonesystem phonesystem 4096 Oct 10 21:54 Bin
drwxr-xr-x  5 phonesystem phonesystem 4096 Jul 24 17:36 Data
drwxr-xr-x  4 phonesystem phonesystem 4096 Jul 24 17:36 Instance1
-rw-r--r--  1 phonesystem phonesystem    2 Jun 27 19:38 VERSION

root@3cx:/var/lib/3cxpbx# cd Instance1/
root@3cx:/var/lib/3cxpbx/Instance1# ls -la

total 16
drwxr-xr-x  4 phonesystem phonesystem 4096 Jul 24 17:36 .
drwxr-xr-x  6 phonesystem phonesystem 4096 Oct  6 04:00 ..
drwxr-xr-x  4 phonesystem phonesystem 4096 Jul 24 17:36 Bin
drwxr-xr-x 13 phonesystem phonesystem 4096 Jul 24 17:36 Data

root@3cx:/var/lib/3cxpbx/Instance1# cd Data
root@3cx:/var/lib/3cxpbx/Instance1/Data# ls -la

total 52
drwxr-xr-x 13 phonesystem phonesystem 4096 Jul 24 17:36 .
drwxr-xr-x  4 phonesystem phonesystem 4096 Jul 24 17:36 ..
drwxr-xr-x  2 phonesystem phonesystem 4096 Oct 12 03:00 Backups
drwxr-xr-x  2 phonesystem phonesystem 4096 Jul 25 16:48 Chat
drwxr-xr-x  2 phonesystem phonesystem 4096 Jun 27 19:52 Documents
drwxr-xr-x  6 phonesystem phonesystem 4096 Jul 24 17:36 Fax
drwxr-xr-x  2 phonesystem phonesystem 4096 Jul 25 16:48 HotelModule
drwxr-xr-x  6 phonesystem phonesystem 4096 Jul 24 17:36 Http
drwxr-xr-x  4 phonesystem phonesystem 4096 Jul 24 17:36 Ivr
drwxr-xr-x  3 phonesystem phonesystem 4096 Oct 12 08:16 Logs
drwxr-xr-x  3 phonesystem phonesystem 4096 Jun 27 19:52 Recordings
drwxr-xr-x  2 phonesystem phonesystem 4096 Jul 25 16:48 Voiceapps
drwxr-xr-x  2 phonesystem phonesystem 4096 Jun 27 19:52 WebSitesTemp

root@3cx:/var/lib/3cxpbx/Instance1/Data# cd Backups/
root@3cx:/var/lib/3cxpbx/Instance1/Data/Backups# ls -la

total 4656916
drwxr-xr-x  2 phonesystem phonesystem      4096 Oct 12 03:00 .
drwxr-xr-x 13 phonesystem phonesystem      4096 Jul 24 17:36 ..
-rw-r--r--  1 phonesystem phonesystem  10227239 Oct 10 22:40 20191010-2240-GeneralBackup-LicenseKeyEtcOnly.zip
-rw-r--r--  1 phonesystem phonesystem 206016591 Oct  5 03:00 3CXScheduledBackup.1.zip
-rw-r--r--  1 phonesystem phonesystem 205880761 Oct  4 03:00 3CXScheduledBackup.2.zip
-rw-r--r--  1 phonesystem phonesystem 204982217 Oct  3 03:00 3CXScheduledBackup.3.zip
-rw-r--r--  1 phonesystem phonesystem 204021514 Oct  2 02:59 3CXScheduledBackup.4.zip
-rw-r--r--  1 phonesystem phonesystem 203622075 Oct  1 03:00 3CXScheduledBackup.5.zip
-rw-r--r--  1 phonesystem phonesystem 203170264 Sep 30 03:00 3CXScheduledBackup.6.zip
-rw-r--r--  1 phonesystem phonesystem 203155436 Sep 29 04:00 3CXScheduledBackup.7.zip
-rw-r--r--  1 phonesystem phonesystem 206038630 Oct  6 02:59 3CXScheduledBackup.zip

root@3cx:/var/lib/3cxpbx/Instance1/Data/Backups#
 
There's many subfolders inside 3cxpbx folder (and where the problem may lie).

Can you check them all please?
 
Hi John,

I ran 'ls -laR' from within the 3cxpbx directory (the ownership and permissions of that directory itself are 'phonesystem' - see above).

It is a long listing, so rather than try to check it manually and potentially missing something, I sent it to a file, opened it in Excel, and filtered by the 'owner' field.

All files and directories are owned by 'phonesystem' with the following exceptions, all of which are owned by 'root':

/var/lib/3cxpbx/Bin/startup
/var/lib/3cxpbx/Data/Logs/unattended-upgrade.log
/var/lib/3cxpbx/Instance1/Data/Logs/dump.pcap

The 'startup' file appears to have been created when the system was first installed.

The 'unattended-upgrade.log' appears to have been created two days ago, at 4am, so it post-dates the backups failing, but it is possible that there was already a version of that file with that same name in the same place, also owned by 'root', before two days ago.

The 'dump.pcap' was created 13 Aug 2019, so it pre-dates the backups failing by about seven weeks.


Are any of those 'unusual' in that you would not expect to see the file(s) and / or would expect them to be owned by 'phonesystem'?

Thanks,

Alan.
 
For the avoidance of doubt, this is what the (unattended-upgrade.log) file contains:

Code:
--2019-10-13 04:00:00--  https://downloads-au.3cx.com/downloads/v160/debianupdate/stretch.txt
Resolving downloads-au.3cx.com (downloads-au.3cx.com)... 35.201.76.132, 2600:1901:0:9ab3::
Connecting to downloads-au.3cx.com (downloads-au.3cx.com)|35.201.76.132|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 23729 (23K) [text/plain]
Saving to: ‘/etc/apt/preferences.d/3cxpbx’

     0K .......... .......... ...                             100%  379K=0.06s

2019-10-13 04:00:01 (379 KB/s) - ‘/etc/apt/preferences.d/3cxpbx’ saved [23729/23729]

Pin file downloaded
Hit:1 http://downloads-global.3cx.com/downloads/debian stretch-testing InRelease
Get:2 http://security.debian.org stretch/updates InRelease [94.3 kB]
Ign:3 http://deb.debian.org/debian stretch InRelease
Get:4 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:5 http://deb.debian.org/debian stretch-backports InRelease [91.8 kB]
Hit:6 http://downloads-global.3cx.com/downloads/debian stretch InRelease
Get:7 http://security.debian.org stretch/updates/main amd64 Packages [500 kB]
Hit:8 http://deb.debian.org/debian stretch Release
Get:9 http://deb.debian.org/debian stretch-backports/main amd64 Packages.diff/Index [27.8 kB]
Get:11 http://deb.debian.org/debian stretch-backports/main amd64 Packages 2019-10-08-0832.59.pdiff [1,262 B]
Get:12 http://deb.debian.org/debian stretch-backports/main amd64 Packages 2019-10-08-1413.44.pdiff [487 B]
Get:12 http://deb.debian.org/debian stretch-backports/main amd64 Packages 2019-10-08-1413.44.pdiff [487 B]
Fetched 807 kB in 1s (509 kB/s)
Reading package lists...
No updates installed.

Look like it checked for updates, found none that were required, and finished without updating anything, but by all means correct me if I am wrong.

Thanks,

Alan.
 
Last edited:
The 3 files mentioned above are ok, no problems in ownership being root.

I suggest scheduling a backup again (using the settings that cause failure) and then read the 3CXSystemService.log. Look for the following info at the correct date and time:

2019/10/15 12:30:00.108|726|0026|Info|Backup started
.
.
.
.
2019/10/15 12:30:10.251|726|0026|Info|Backup finished

Then read the rest of the lines between those two info tags to see if you can catch any errors that specifically tell you what failed.
 
Status
Not open for further replies.

Forum statistics

Threads
111,934
Messages
589,818
Members
164,811
Latest member
aurorasigntrtechitnet