Toggle sidebar Toggle sidebar

Solved Backup Failure V16.0.5.612

Status
Not open for further replies.

[email protected]

Customer
Basic Certified
Joined
Aug 21, 2014
Messages
20
Reaction score
0
Howdy!

I am trying to run a backup so I can move the PBX to new hardware. Based on the scheduled backups, which run weekly the file size should be 2gb ish. When I run a manual backup it seems to stop around 87meg. It's been around 8 hours, so it should be finished by now.

Is there a log or anything I can look at? Nothing in the Windows Application log related to postgres or any services stopping. Nothing in the Activity log that looked out of place either. Any pointers?
 
Hi,

Normally you will receive an email that the backup has been completed.

Has the email arrived for the smaller backup?

Did you perhaps not tick additional options during the manual backkup? That would result in a significantly smaller size (no recordings, prompts, firmware, voicemails were included)
 
Howdy! No emails saying it was complete. I ticked everything as I need to move the install to another hardware platform. Everything else appears to be working. I even did a reboot of the PBX before I did that backup. I wanted to make sure it completed without a hiccup.

Is there a log or a way to see if the job is hung?
 
If it's taking that long to run, you are probably using google drive storage (deprecated) or some other service that is very slow. If you have space on the PBX's existing server, change the backup to local and run it again. The backups are always faster when they don't have to transfer the data elsewhere.
 
Currently I am using the local disk as storage. I see the file grow to a certain point then stop in the following folder:

C:\ProgramData\3CX\Instance1\Data\Backups

I extracted the one that doesn't appear to complete and it looks like it has most of the data, except voicemails. It has VM for 1 extension, that's it. So it might be hanging on the voicemail part. Looking at the Voicemail folders, those are about 1gb. So it should definitely be larger then it seems to be.
 
Go to Activity log, Settings, then change the logs to verbose.

Restart your services and run a full backup.

Once you notice the activity has stopped, check the Management Console log.

You can search for the word backup around the the approximate timeframe
 
Do you have any AV running on the server? Exempt that backup folder from scans. Also, don't save recordings just to test a new backup job. Thanks.
 
JohnS_3CX said:
Go to Activity log, Settings, then change the logs to verbose.

Restart your services and run a full backup.

Once you notice the activity has stopped, check the Management Console log.

You can search for the word backup around the the approximate timeframe
Click to expand...

Where is the Management Console Log? Is that the Activity Log from the Dashboard, or something else?
 
You will find it under

Dashboard > Activity Log > Click Logs Button on top > Instance
 
WOW. Never explored that. :)

This is the first message thrown. Lots of like it. I'll send you the log.

2020/05/13 09:20:12.091|11416|0017|Exc|System.AggregateException: One or more errors occurred. (Exception while reading from stream)
---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
at Npgsql.AwaitableSocket.GetResult()
at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlRawCopyStream.Read(Byte[] buffer, Int32 offset, Int32 count, Boolean async)
at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.BackupTableAsync(String fromTable, Stream outputStream, Object transactionObject)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.<>c__DisplayClass13_0.<BackupTable>b__0(Stream x)
 
Window Defender is the AV, nothing fancy on the box.

I see a few more exceptions, then see the final event. The job was called FullBackupSupport

2020/05/13 09:20:12.114|11416|0024|Exc|System.AggregateException: One or more errors occurred. (This NpgsqlTransaction has completed; it is no longer usable.)
---> System.InvalidOperationException: This NpgsqlTransaction has completed; it is no longer usable.
at Npgsql.NpgsqlTransaction.CheckCompleted()
at Npgsql.NpgsqlTransaction.CheckReady()
at Npgsql.NpgsqlTransaction.Commit(Boolean async)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.CommitAsync(Object transactionObject)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.RunAll(BaseTask ignore)
2020/05/13 09:20:12.114|11416|0024|Exc|System.AggregateException: One or more errors occurred. (This NpgsqlTransaction has completed; it is no longer usable.)
---> System.InvalidOperationException: This NpgsqlTransaction has completed; it is no longer usable.
at Npgsql.NpgsqlTransaction.CheckCompleted()
at Npgsql.NpgsqlTransaction.CheckReady()
at Npgsql.NpgsqlTransaction.Commit(Boolean async)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.CommitAsync(Object transactionObject)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.RunAll(BaseTask ignore)
2020/05/13 09:20:12.114|11416|0024|Err|System.AggregateException: One or more errors occurred. (This NpgsqlTransaction has completed; it is no longer usable.)
---> System.InvalidOperationException: This NpgsqlTransaction has completed; it is no longer usable.
at Npgsql.NpgsqlTransaction.CheckCompleted()
at Npgsql.NpgsqlTransaction.CheckReady()
at Npgsql.NpgsqlTransaction.Commit(Boolean async)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.CommitAsync(Object transactionObject)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.RunAll(BaseTask ignore)
2020/05/13 09:20:12.114|11416|0024|Inf|Saving MyPhone History: state=Cancelling
2020/05/13 09:20:12.114|11416|0024|Inf|Saving Statistics tables: state=Cancelling
2020/05/13 09:20:12.114|11416|0024|Inf|Saving http Prompts: state=Cancelling
2020/05/13 09:20:12.114|11416|0024|Inf|Saving IVR Prompts: state=Cancelling
2020/05/13 09:20:12.114|11416|0024|Inf|Saving Voicemail prompts: state=Cancelling
2020/05/13 09:20:12.114|11416|0024|Inf|Saving Phone provisioning files: state=Cancelling
2020/05/13 09:20:12.114|11416|0024|Inf|Task Saving Call History was processed for 21.1741738 seconds.
2020/05/13 09:20:12.122|11416|0021|Inf|Saving http Prompts: state=Cancelled
2020/05/13 09:20:15.172|11416|0025|Inf|Table meetingsession was zipped for: 3049 ms
2020/05/13 09:20:15.172|11416|0025|Inf|Task Saving Statistics tables was processed for 24.2305742 seconds.
2020/05/13 09:20:15.172|11416|0005|Inf|Saving Voicemail prompts: state=Cancelled
2020/05/13 09:20:15.175|11416|0027|Inf|Saving IVR Prompts: state=Cancelled
2020/05/13 09:20:15.175|11416|0020|Inf|Saving Phone provisioning files: state=Cancelled
2020/05/13 09:20:15.176|11416|0030|Inf|Starting finishing task...
2020/05/13 09:20:15.176|11416|0030|Inf|Backup FullBackupSupport created
2020/05/13 09:20:15.178|11416|0030|Inf|Email about backup failure will not be sent, NOTIFY_BACKUP_FAILED=0
.
 
Did you end up getting a resolution? What is 3CX running on, windows? What version and patch level? Any other services or functions on it?
 
No resolution as of yet. AV excluded and disabled. Windows 10 1909, current cu. No other role. We just need to get a backup so we can move it to it's replacement hardware. :) I have done this a bunch of times on others machines and haven't had an issue. Kinda surprised on this one. I am open to any and all ideas. :)
 
Go ahead and disable VSS service and make sure you suspend any third party backup software. How do you do image or system backups? Restart machines after you disable and try again. Physical or virtual machine? Thanks.
 
I forgot to ask what AV are you running? I would try uninstalling AV as well.
 
AV is Windows Defender. No Backup. Nothing else 3rd party besides Chrome and Edge (New Version). No disk issues, Windows Event Logs look clean.
 
Didn't think of using the cmdline backup. I like that.

BackupCmd.exe --f=supportbackup.zip --l supportbackup.log --options=ALL

Same type of error in the supportbackup.log file.This is the complete failure part of the log.

Any ideas?

2020/05/13 13:54:56.368|516|0008|Info|Table cl_segments was zipped for: 6328 ms
2020/05/13 13:54:56.369|516|0008|Info|Task Saving Call History was processed for 32.9726182 seconds.
2020/05/13 13:54:56.369|516|0008|Info|Saving Call History: state=Finished
2020/05/13 13:54:58.044|516|0011|Excpt|System.AggregateException: One or more errors occurred. (Exception while reading from stream)
---> Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
at Npgsql.AwaitableSocket.GetResult()
at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Npgsql.NpgsqlRawCopyStream.Read(Byte[] buffer, Int32 offset, Int32 count, Boolean async)
at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.BackupTableAsync(String fromTable, Stream outputStream, Object transactionObject)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.<>c__DisplayClass13_0.<BackupTable>b__0(Stream x)
2020/05/13 13:54:58.044|516|0011|Warn|Skip backup table myphone_callhistory_v14
2020/05/13 13:54:58.045|516|0011|Info|Table myphone_callhistory_v14 was zipped for: 1676 ms
2020/05/13 13:54:58.046|516|0011|Excpt|System.AggregateException: One or more errors occurred. (Exception while writing to stream)
---> Npgsql.NpgsqlException (0x80004005): Exception while writing to stream
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
at Npgsql.AwaitableSocket.GetResult()
at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)
at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)
at Npgsql.NpgsqlConnector.ExecuteInternalCommand(Byte[] data, Boolean async)
at Npgsql.NpgsqlTransaction.Commit(Boolean async)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.CommitAsync(Object transactionObject)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.RunAll(BaseTask ignore)
2020/05/13 13:54:58.047|516|0011|Excpt|System.AggregateException: One or more errors occurred. (Exception while writing to stream)
---> Npgsql.NpgsqlException (0x80004005): Exception while writing to stream
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
at Npgsql.AwaitableSocket.GetResult()
at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)
at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)
at Npgsql.NpgsqlConnector.ExecuteInternalCommand(Byte[] data, Boolean async)
at Npgsql.NpgsqlTransaction.Commit(Boolean async)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.CommitAsync(Object transactionObject)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.RunAll(BaseTask ignore)
2020/05/13 13:54:58.047|516|0011|Erro|System.AggregateException: One or more errors occurred. (Exception while writing to stream)
---> Npgsql.NpgsqlException (0x80004005): Exception while writing to stream
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
at Npgsql.AwaitableSocket.GetResult()
at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)
at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)
at Npgsql.NpgsqlConnector.ExecuteInternalCommand(Byte[] data, Boolean async)
at Npgsql.NpgsqlTransaction.Commit(Boolean async)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.CommitAsync(Object transactionObject)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.RunAll(BaseTask ignore)
2020/05/13 13:54:58.048|516|0011|Info|Saving MyPhone History: state=Cancelling
2020/05/13 13:54:58.048|516|0011|Info|Saving http Prompts: state=Cancelling
2020/05/13 13:54:58.048|516|0011|Info|Saving IVR Prompts: state=Cancelling
2020/05/13 13:54:58.048|516|0011|Info|Saving Voicemail prompts: state=Cancelling
2020/05/13 13:54:58.048|516|0011|Info|Saving Voice mails: state=Cancelling
2020/05/13 13:54:58.048|516|0011|Info|Saving Phone provisioning files: state=Cancelling
2020/05/13 13:54:58.048|516|0011|Info|Task Saving MyPhone History was processed for 34.0532091 seconds.
2020/05/13 13:54:58.054|516|0018|Info|Saving Voicemail prompts: state=Cancelled
2020/05/13 13:54:58.068|516|0016|Info|Saving http Prompts: state=Cancelled
2020/05/13 13:54:58.071|516|0017|Info|Saving IVR Prompts: state=Cancelled
2020/05/13 13:54:58.083|516|0019|Info|Saving Voice mails: state=Cancelled
2020/05/13 13:54:58.083|516|0022|Info|Saving Phone provisioning files: state=Cancelled
2020/05/13 13:54:58.084|516|0024|Info|Starting finishing task...
2020/05/13 13:54:58.086|516|0001|Info|Email about backup failure will not be sent, NOTIFY_BACKUP_FAILED=0
 
BTW can I edit the thread title? I didn't realize I didn't finish it out. oops.
 
Status
Not open for further replies.

Latest Posts

Forum statistics

Threads
111,948
Messages
589,880
Members
164,841
Latest member
erre
3CX Logo Footer Copyright © Sitemap Privacy
Top Bottom