Solved Scheduled Backup Failure

Discussion in '3CX Phone System - General' started by venkman, Feb 2, 2017.

Thread Status:
Not open for further replies.
  1. venkman

    Joined:
    Jul 22, 2015
    Messages:
    21
    Likes Received:
    0
    Hi,

    We have just started getting 3CX scheduled backup failures, two days in a row. When I run a manual backup, it completes with no issues.

    Here is the error I get when the backup fails:

    3CX Phone System tried to backup but failed due to the following reason(s):


    System.Exception: Error creating backup ---> System.AggregateException: One or more errors occurred. ---> Npgsql.NpgsqlException: The connection pool has been exhausted, either raise MaxPoolSize (currently 100) or Timeout (currently 600 seconds)


    at Npgsql.ConnectorPool.<WaitForTaskAsync>d__24.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Npgsql.ConnectorPool.<AllocateAsync>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Npgsql.NpgsqlConnection.<OpenInternalAsync>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.<CreateConnectionAsync>d__26.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.<BackupTableAsync>d__27.MoveNext()
    --- End of inner exception stack trace ---
    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
    at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.<>c__DisplayClass13_0.<BackupTable>b__0(Stream x)
    at _3cxNetBackupAndRestore._3cxBackup.ZipDbTable(String table, Action`1 writeContent)
    at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.<RunAll>d__12.MoveNext()
    --- End of inner exception stack trace ---
    at _3cxNetBackupAndRestore._3cxBackup.WaitForFinish()
    at _3cxBackupRestoreImpl.BackupRestoreUtils.Backup(String instanceId, String cfgPath, String dbConnectionString, BackupConfig options, String fullpathFilename, NetworkCredential credentials, Boolean sendEmail)
    at BackupRestoreSchedule.BackupJob.Execute(IJobExecutionContext context)


    I haven't attempted a reboot of the server yet as I'd like to know more about the error before possibly resolving with a reboot. i.e. Does any know what could be causing this?

    3CX is running on a Win2012R2 VM, 8GB RAM 2vCPU and there is plenty of space on the local drive where the backups reside. Running 3CX Pro 15.0.60903.0.
     
  2. GiannosC_3CX

    GiannosC_3CX Guest

    Hi Venkman,

    Please disable your schedule backup, next restart your services and then enable it and let us know if this error still received.
     
  3. eagle2

    eagle2 Well-Known Member

    Joined:
    Apr 27, 2011
    Messages:
    1,085
    Likes Received:
    11
    I got also an error with scheduled backup on Debian, after restart of services, backup (not scheduled) is OK.

    3CX Phone System tried to backup but failed due to the following reason(s):
    System.Exception: Error creating backup ---> System.AggregateException: One or more errors occurred. (53300: sorry, too many clients already) ---> Npgsql.PostgresException: 53300: sorry, too many clients already
    at Npgsql.NpgsqlConnector.<DoReadMessageAsync>d__6.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Npgsql.NpgsqlConnector.<ReadMessageWithPrependedAsync>d__5.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Npgsql.NpgsqlConnector.<HandleAuthenticationAsync>d__2.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Npgsql.NpgsqlConnector.<OpenAsync>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Npgsql.ConnectorPool.<AllocateAsync>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Npgsql.NpgsqlConnection.<OpenInternalAsync>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.<CreateConnectionAsync>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.<BeginTransactionAsync>d__10.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at _3cxNetInterfaces.Implementation.Npgsql.NpgsqlDbOperation.<BeginTransactionAsync>d__9.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at _3cxNetBackupAndRestore.BackupTasks.BaseDBTask.<RunAll>d__12.MoveNext()
    --- End of inner exception stack trace ---
    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
    at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
    at _3cxNetBackupAndRestore.BackupTasks.MyPhoneTask.<.ctor>b__0_0(BaseTask x)
    at _3cxNetManagement.BaseTask.Run()
    --- End of inner exception stack trace ---
    at _3cxNetBackupAndRestore._3cxBackup.WaitForFinish()
    at _3cxBackupRestoreImpl.BackupRestoreUtils.Backup(String instanceId, String cfgPath, String dbConnectionString, BackupConfig options, String fullpathFilename, NetworkCredential credentials, Boolean sendEmail)
    at BackupRestoreSchedule.BackupJob.Execute(IJobExecutionContext context)

     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. phil@lgmanor.com

    Joined:
    Oct 13, 2016
    Messages:
    25
    Likes Received:
    2
    I started another thread before I saw this, but having the exact same issue. Just started 2 days ago. Manual backup works correctly.
     
  5. datamerge

    datamerge New Member

    Joined:
    Nov 19, 2014
    Messages:
    169
    Likes Received:
    19
    I am having this issue too. I raised a case on it. As part of the fault finding I was asked to Wireshark the activity. So I changed the time to 5 minutes ahead and then sat in anticipation. The backup did not fire, so I went home. By the time I got home the backup had gone off on half hour after my scheduled time. (I scheduled 20:18 and it fired at 20:30). It worked! I thought that was it, but then the next day it failed at 20:30.

    Mine is Linux version.
     
  6. venkman

    Joined:
    Jul 22, 2015
    Messages:
    21
    Likes Received:
    0
    We were advised to reboot the server by our support provider, and the scheduled backup completed successfully (as I suspected after a reboot). Would still be interested to find out what the route cause of this was.
     
  7. venkman

    Joined:
    Jul 22, 2015
    Messages:
    21
    Likes Received:
    0
    As part of my reboot last night Windows Updates KB3205404, KB3205401 & KB2538243 were installed. I doubt that this was the resolution, but still might be worth checking that you have no outstanding Windows Updates to apply?
     
  8. Sopock

    Sopock Member

    Joined:
    Jul 11, 2012
    Messages:
    448
    Likes Received:
    20
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. datamerge

    datamerge New Member

    Joined:
    Nov 19, 2014
    Messages:
    169
    Likes Received:
    19
    @venkman. When you say your manual backup is OK, have you checked the backup file to make sure it looks valid? I.E. Is it about the same size as your scheduled backup files if you select the same options?

    I have found that the manual backup always reports it worked OK, even when it doesn't. I suspect your manual backup is failing too.

    The error message "Npgsql.PostgresException: 53300: sorry, too many clients already" brings many hits on Google and it is the result of too many connections being made concurrently to the Postgres database. It appear that some component of 3CX may not be closing connections after some operation(s) and eventually the concurrent connection count is exceeded. Rebooting the 3CX resolves the issue. I haven't tried it yet, but I suspect restarting all 3CX services would also fix it.
     
  10. venkman

    Joined:
    Jul 22, 2015
    Messages:
    21
    Likes Received:
    0
    Thanks for the response datamerge. As per my previous reply above, a restart of the server resolved this.
     
Thread Status:
Not open for further replies.