• V20: 3CX Re-engineered. Get V20 for increased security, better call management, a new admin console and Windows softphone. Learn More.

Google Drive Backup - Rate Limit Exceeded [403]

Status
Not open for further replies.

François Lepage

Customer
Joined
Sep 23, 2018
Messages
33
Reaction score
2
Hi,

During the last 2 weeks approximately, the scheduled backup for my system have failed with the following error.

The service drive has thrown an exception: Google.GoogleApiException: Google.Apis.Requests.RequestError Rate Limit Exceeded [403]

This error happens randomly. Sometimes, the backup does go through.

I'm no expert of the Google Drive API but this sounds like some kind on throttling - or rate limiting on Google's end.

I'm running 15.5.15502.6 and G-Suite Business.

Here is the full stack trace :

The service drive has thrown an exception: Google.GoogleApiException: Google.Apis.Requests.RequestError
Rate Limit Exceeded [403]
Errors [
Message[Rate Limit Exceeded] Location[ - ] Reason[rateLimitExceeded] Domain[usageLimits]
]

at Google.Apis.Requests.ClientServiceRequest`1.<ParseResponse>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Google.Apis.Requests.ClientServiceRequest`1.Execute()
at Google.Apis.Requests.PageStreamer`4.<Fetch>d__5.MoveNext()
at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at GoogleDriveImpl.GdFolder.DoGetFiles(String type)
at GoogleDriveImpl.GdFolder.GetFolderId(String path)
at GoogleDriveImpl.GdFolder..ctor(GoogleDriveCreds creds, String path, String folderId)
at _3cxBackupRestoreImpl.Filesystem.GoogleDrive.GdFileSystem.Delete(String filename)
at _3cxBackupRestoreImpl.BackupRestoreUtils.Backup(IFilesystem filesystem, String instanceId, String cfgPath, String dbConnectionString, BackupConfig options, String fullpathFilename, Boolean sendEmail)
at BackupRestoreSchedule.BackupJob.Execute(IJobExecutionContext context)

Thanks for your help

François
 
We get the same error, if I go in and do a manual back up it goes through OK.
 
We also get that error once in a while. Let's say that about 10% fails this way.
 
You are right, it is Google rate limiting, and I'm fairly certain googling the error will direct you to steps to increase the limits.
 
Anyone got stats on what they are allowed / vs what they have?
 
+1 – About one in every eight fails, we are not using the API for anything else...
 
Hi,

During the last 2 weeks approximately, the scheduled backup for my system have failed with the following error.

The service drive has thrown an exception: Google.GoogleApiException: Google.Apis.Requests.RequestError Rate Limit Exceeded [403]

This error happens randomly. Sometimes, the backup does go through.

I'm no expert of the Google Drive API but this sounds like some kind on throttling - or rate limiting on Google's end.

I'm running 15.5.15502.6 and G-Suite Business.

Here is the full stack trace :

The service drive has thrown an exception: Google.GoogleApiException: Google.Apis.Requests.RequestError
Rate Limit Exceeded [403]
Errors [
Message[Rate Limit Exceeded] Location[ - ] Reason[rateLimitExceeded] Domain[usageLimits]
]

at Google.Apis.Requests.ClientServiceRequest`1.<ParseResponse>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Google.Apis.Requests.ClientServiceRequest`1.Execute()
at Google.Apis.Requests.PageStreamer`4.<Fetch>d__5.MoveNext()
at System.Collections.Generic.List`1.AddEnumerable(IEnumerable`1 enumerable)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at GoogleDriveImpl.GdFolder.DoGetFiles(String type)
at GoogleDriveImpl.GdFolder.GetFolderId(String path)
at GoogleDriveImpl.GdFolder..ctor(GoogleDriveCreds creds, String path, String folderId)
at _3cxBackupRestoreImpl.Filesystem.GoogleDrive.GdFileSystem.Delete(String filename)
at _3cxBackupRestoreImpl.BackupRestoreUtils.Backup(IFilesystem filesystem, String instanceId, String cfgPath, String dbConnectionString, BackupConfig options, String fullpathFilename, Boolean sendEmail)
at BackupRestoreSchedule.BackupJob.Execute(IJobExecutionContext context)

Thanks for your help

François

I would suggest you look into your backup setup how big is your retention policy? The smaller the retention the better you are able to handle your limited cloud space.

Just as a reminder just double check manually to see how much space is remaining on the account that you are using just to verify that you are near your storage cap.

I got this recently and I noticed that I thought it was a great idea to have a backup retention of 6 but that was too much.

Might help.
 
I have an unlimited google drive subscription (old Google subscription). I have a daily limit but I do not exceed it.

I have same problem.. randomly
 
We are seeing this as well. Very random, maybe 5% of nightly scheduled backups.
 
I went back on a batch of about 20 servers sharing same Google Drive and adjusted backup times so they did not overlap. 1st night, the first two failed with this error at 6 & 7pm, all after succeeded. 2nd night the 6th one failed at 8pm, all before and after succeeded.
 
From google REST API:

The user has reached Google Drive API's maximum request rate. The limit varies depending on the kind of requests.

Suggested actions:
 
From google REST API:

The user has reached Google Drive API's maximum request rate. The limit varies depending on the kind of requests.

Suggested actions:

I have seen that before as well, which seems to indicate to me it's something within the actions/api calls within the single job itself, not a limit we've reached because of space, too many jobs, daily limits, or overlapping times etc.
 
@bbaker73 you are correct, this is something within the way 3CX handles the Rest API. Nothing to do with the space you are using / have left, or the amount of jobs you are sending. They have to throttle it within the app.
 
  • Like
Reactions: bbaker73
We have this problem some weeks ago on multiples customers installations.
The 3CX support tell us this is a google's API issue. I was very supprised because we re using one google account for each customer and not every installation was concern.

I tried to change hour (4 am, 11 pm) in the backup schedule and the problem don't come back again... Give a try to this "workaround".
 
We have this problem some weeks ago on multiples customers installations.
The 3CX support tell us this is a google's API issue. I was very supprised because we re using one google account for each customer and not every installation was concern.

I tried to change hour (4 am, 11 pm) in the backup schedule and the problem don't come back again... Give a try to this "workaround".

Anythings worth a shot I suppose, but I've had errors at 6pm, 7, 8, 10, 11, 12, 1am, 2am, 3am etc.
 
I changed our backup schedule from weekly to daily at 2 am as we were getting around a 50% failure rate, we have now had 7 failures in 8 days!
Yet if I do a manual backup to the same location using the same parameters I have a 100% success rate???
I've changed the backup time in the hope that it will improve the success rate.
 
Just to add we have changed the back up time from 2 am to mid morning and touch wood we have had a 100% success rate.
 
Status
Not open for further replies.

Getting Started - Admin

Latest Posts

Forum statistics

Threads
141,630
Messages
748,955
Members
144,746
Latest member
gamingpro2131
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.