• V20: 3CX Re-engineered. Get V20 for increased security, better call management, a new admin console and Windows softphone. Learn More.

Solved Scheduled Backup Failure

Status
Not open for further replies.

venkman

Joined
Jul 22, 2015
Messages
20
Reaction score
0
Hi,

We have just started getting 3CX scheduled backup failures, two days in a row. When I run a manual backup, it completes with no issues.

Here is the error I get when the backup fails:

3CX Phone System tried to backup but failed due to the following reason(s):


System.Exception: Error creating backup ---> System.AggregateException: One or more errors occurred. ---> Npgsql.NpgsqlException: The connection pool has been exhausted, either raise MaxPoolSize (currently 100) or Timeout (currently 600 seconds)


at Npgsql.ConnectorPool.<WaitForTaskAsync>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.ConnectorPool.<AllocateAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnection.<OpenInternalAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.<CreateConnectionAsync>d__26.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.<BackupTableAsync>d__27.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.<>c__DisplayClass13_0.<BackupTable>b__0(Stream x)
at _3cxNetBackupAndRestore._3cxBackup.ZipDbTable(String table, Action`1 writeContent)
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.<RunAll>d__12.MoveNext()
--- End of inner exception stack trace ---
at _3cxNetBackupAndRestore._3cxBackup.WaitForFinish()
at _3cxBackupRestoreImpl.BackupRestoreUtils.Backup(String instanceId, String cfgPath, String dbConnectionString, BackupConfig options, String fullpathFilename, NetworkCredential credentials, Boolean sendEmail)
at BackupRestoreSchedule.BackupJob.Execute(IJobExecutionContext context)


I haven't attempted a reboot of the server yet as I'd like to know more about the error before possibly resolving with a reboot. i.e. Does any know what could be causing this?

3CX is running on a Win2012R2 VM, 8GB RAM 2vCPU and there is plenty of space on the local drive where the backups reside. Running 3CX Pro 15.0.60903.0.
 
Hi Venkman,

Please disable your schedule backup, next restart your services and then enable it and let us know if this error still received.
 
I got also an error with scheduled backup on Debian, after restart of services, backup (not scheduled) is OK.

3CX Phone System tried to backup but failed due to the following reason(s):
System.Exception: Error creating backup ---> System.AggregateException: One or more errors occurred. (53300: sorry, too many clients already) ---> Npgsql.PostgresException: 53300: sorry, too many clients already
at Npgsql.NpgsqlConnector.<DoReadMessageAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<ReadMessageWithPrependedAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<HandleAuthenticationAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnector.<OpenAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.ConnectorPool.<AllocateAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Npgsql.NpgsqlConnection.<OpenInternalAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.<CreateConnectionAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.<BeginTransactionAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.<BeginTransactionAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.<RunAll>d__12.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at _3cxNetBackupAndRestore.BackupTasks.MyPhoneTask.<.ctor>b__0_0(BaseTask x)
at _3cxNetManagement.BaseTask.Run()
--- End of inner exception stack trace ---
at _3cxNetBackupAndRestore._3cxBackup.WaitForFinish()
at _3cxBackupRestoreImpl.BackupRestoreUtils.Backup(String instanceId, String cfgPath, String dbConnectionString, BackupConfig options, String fullpathFilename, NetworkCredential credentials, Boolean sendEmail)
at BackupRestoreSchedule.BackupJob.Execute(IJobExecutionContext context)

 
I started another thread before I saw this, but having the exact same issue. Just started 2 days ago. Manual backup works correctly.
 
I am having this issue too. I raised a case on it. As part of the fault finding I was asked to Wireshark the activity. So I changed the time to 5 minutes ahead and then sat in anticipation. The backup did not fire, so I went home. By the time I got home the backup had gone off on half hour after my scheduled time. (I scheduled 20:18 and it fired at 20:30). It worked! I thought that was it, but then the next day it failed at 20:30.

Mine is Linux version.
 
We were advised to reboot the server by our support provider, and the scheduled backup completed successfully (as I suspected after a reboot). Would still be interested to find out what the route cause of this was.
 
I started another thread before I saw this, but having the exact same issue. Just started 2 days ago. Manual backup works correctly.

As part of my reboot last night Windows Updates KB3205404, KB3205401 & KB2538243 were installed. I doubt that this was the resolution, but still might be worth checking that you have no outstanding Windows Updates to apply?
 
@venkman. When you say your manual backup is OK, have you checked the backup file to make sure it looks valid? I.E. Is it about the same size as your scheduled backup files if you select the same options?

I have found that the manual backup always reports it worked OK, even when it doesn't. I suspect your manual backup is failing too.

The error message "Npgsql.PostgresException: 53300: sorry, too many clients already" brings many hits on Google and it is the result of too many connections being made concurrently to the Postgres database. It appear that some component of 3CX may not be closing connections after some operation(s) and eventually the concurrent connection count is exceeded. Rebooting the 3CX resolves the issue. I haven't tried it yet, but I suspect restarting all 3CX services would also fix it.
 
@venkman. When you say your manual backup is OK, have you checked the backup file to make sure it looks valid? I.E. Is it about the same size as your scheduled backup files if you select the same options?

I have found that the manual backup always reports it worked OK, even when it doesn't. I suspect your manual backup is failing too.

The error message "Npgsql.PostgresException: 53300: sorry, too many clients already" brings many hits on Google and it is the result of too many connections being made concurrently to the Postgres database. It appear that some component of 3CX may not be closing connections after some operation(s) and eventually the concurrent connection count is exceeded. Rebooting the 3CX resolves the issue. I haven't tried it yet, but I suspect restarting all 3CX services would also fix it.

Thanks for the response datamerge. As per my previous reply above, a restart of the server resolved this.
 
Status
Not open for further replies.

Getting Started - Admin

Latest Posts

Forum statistics

Threads
141,632
Messages
748,964
Members
144,751
Latest member
TECHXNEPAL
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.