Solved GCS archiving and failure conditions

[Willvdl]

We've been using Google bucket archiving features for years but recently have had a scenario where call recordings went missing.
Our managed service provider pointed the finger at the archiving action, saying that Google could have lost the archive during the write.
Which leads me to the question, does 3CX give any indication as to the success or failure of a (scheduled) archiving action?

i.e. what if the archive write fails? How would we know?
I'm not asking about the integrity of the archive solution itself, it is Google job not to lose its files, I'm asking about the physical writing of data from 3CX to the archive. If that write fails, then how would we know?

Many thanks,
Will

 

[VasilisV_3CX]

Hi @will.vanderleij@episodic.,

If the backup is a scheduled backup, you can check the 3cxsystemservice.log for something like:

2022/02/09 10:37:01.609|4113|0052|Trac|Archive recording 101/[example]_109-101_20210604074553(5).wav
2022/02/09 10:37:01.609|4113|0052|Info|101/[example]_109-101_20210604074553(5).wav was successfully archived

Also just to note, v18 includes the new Audit Log feature. If you suspect someone deleted the recordings and you're running v18, you can go to it via the Dashboard and see who did what, when.

 

[Willvdl]

Thank you. MY service provider told me that those logs don't exist prior to version 18

 

[VasilisV_3CX]

Dankie vir jou vinnige antwoord @will.vanderleij@episodic.,

Is this backup a scheduled backup? How often does it run? Do the recordings exist on the system at the moment and they weren't archived, or during the archiving process all of the recordings went missing? Can your provider confirm that they do not exist in the folder, via SSH?

 

[Willvdl]

It is a scheduled archive to a GCS (google bucket) with various compliance holds.
It runs daily because I'd rather have my data there with eleven 9's than on someone else's NAS.

So recordings disappeared for 2 random days in the past. Full 24 hours of recordings missing.
Service provider says:

• once archived, recordings removed from disk - I get that, it's the whole point of an archive
• because of that they can't show that recordings did exist prior to archiving
• They said those kind of logs don't exist in 3CX

So I can't get proof that a recording was made, nor proof that an archiving action failed and I'm left with the blame.

 

[VasilisV_3CX]

Sure @will.vanderleij@episodic.,

When did this happen last? You mention no logs prior to v18, but you've been on v18 for a while now?

Important: Do the archived recordings show in the Management Console under recordings? You should see the recordings with the archived icon next to them. If you don't see them, there is a good chance that they've been removed from the database and that implies that someone deleted them or some other process removed them.

Something to note is that 3CX won't delete the recordings unless the archive was successful.

 

[Willvdl]

Thank you. tHis is what I suspected.
The last known data loss was 2021/07/02 (there may be others but I'd have to and search by hand!)
There are no recording icons for that day at all which tells me, as you say, that either they were deleted or they never happened.
I can't prove it one way or the other as I don't have access to server logs from the service provider.
The situation is, well, rather untenable

 

[VasilisV_3CX]

I understand, this would be something you'd need to discuss with your partner. What I've told you above is information you can use to perhaps find out what happened. As it happened so long ago, I don't think there will be much hope of having any log file that can provide you with the confirmation that you're looking for.

If there is anything else we can do for you, feel free to let me know.

 

[Willvdl]

Thanks. What I needed to confirm is that an Archiving action will appear in a log somewhere for success or failure.

Service provider told me that that is not possible so they waive responsibility as the archive location is not theirs.
I contested that saying that we should at least know if an archived failed so that we can remediate.

This is very useful for me, thanks.