Office Hours Disappearing

[jsabean]

I work for an MSP and manage quite a few 3CX systems. Two days ago one of our clients had some servers go down, and when I called them their phones went to after hours during the afternoon, so I checked 3CX and their office hours had all been wiped out. I checked the audit logs and they were blank for the past month. I was trying to figure out who did it when yesterday we had a 2nd client email us that their phones weren't working. I logged into the management console and checked office hours, they were wiped out as well, only this client's system still had audit logs but no one had connected to their system since 5/23 and the only changes to their business hours were the ones I made yesterday to get the phones back operational.

So now I'm wondering if anyone else has seen any issues lately since this has happened to two of our clients in two days, I don't think that's a coincidence.

• 3CX Version, Enterprise Annual v.18.0 Update 3 (Build 461) on both systems
• Server OS Debian 9 on both systems
• Is the 3CX Server Hosted and where? Hosted in AWS for both systems
• Trunk Provider or VoIP Gateway Make/Model, Bandwidth for both systems
• Are custom Phone Templates being used: No

Thanks!

 

[Alphabetic]

Cant say i've experienced this..

though, the OS was updated to Debian 10 in v18. I would recommend you reinstall using the 3CX ISO.

 

[jsabean]

Thanks, I think I fat fingered that one putting 9 on it, it's on 10

 

[JohnS_3CX]

one of our clients had some servers go down

I would start by investigating what "servers go down" means, and how it was detected as down. This is the most important thing right now - collecting evidence

Some ideas:

- Simple things: Were the office hours perhaps only defined on the trunk or only on specific extensions?

- Simple things: Perhaps someone edited the settings at some point?

- Simple things: Are we looking at the same machine or is there a duplicate machine somewhere out there resolving to the same FQDN?

- Advanced: Perhaps snapshots were restored? This would leave other evidence behind (ie. call logs or recordings will have a gap of days from the last snapshot, chats would be missing etc)

- Advanced: Perhaps a 3CX backup was restored? The filesystem would also show evidence of files being written with no date gaps, yet the 3CX software would be missing data if it used a 3CX Restore operation recently (any warning emails received?)



MSPs usually employ different methods of backup, redundancy or duplication. They usually have more than one person accessing/managing the VMs so you might be looking at admin error, causing those systems to revert to some previous state by mistake. You need to dig for more evidence and see where it leads you.

 

[jsabean]

I would start by investigating what "servers go down" means, and how it was detected as down. This is the most important thing right now - collecting evidence

Some ideas:

- Simple things: Were the office hours perhaps only defined on the trunk or only on specific extensions?

- Simple things: Perhaps someone edited the settings at some point?

- Simple things: Are we looking at the same machine or is there a duplicate machine somewhere out there resolving to the same FQDN?

- Advanced: Perhaps snapshots were restored? This would leave other evidence behind (ie. call logs or recordings will have a gap of days from the last snapshot, chats would be missing etc)

- Advanced: Perhaps a 3CX backup was restored? The filesystem would also show evidence of files being written with no date gaps, yet the 3CX software would be missing data if it used a 3CX Restore operation recently (any warning emails received?)



MSPs usually employ different methods of backup, redundancy or duplication. They usually have more than one person accessing/managing the VMs so you might be looking at admin error, causing those systems to revert to some previous state by mistake. You need to dig for more evidence and see where it leads you.

Thanks for your response. Servers go down was just the short version and unrelated, that client ended up having a power outage for their building. We host their PBX on AWS so other than the VOIP phones not having power that wouldn't have affected the system.

I was definitely leaning toward admin error as some of our clients maintain parts of their system, and Office Hours are one of the things they manage, and since the audit log was blank that was what I was guessing happened and was going to write it off as human error until the 2nd client had the same thing happen and there was nothing in the audit logs showing anyone even logged into their system for 3 days.

I checked the trunks, and no office hours were set up there either on either client, that was one of the first things I checked thinking the same that you did that maybe they just had it set up differently but there were no hours anywhere on either client and when I added them to the Global Office Hours their phones started working again immediately in both situations.

There aren't duplicate machines, as these are both AWS hosted systems.

I'll dig into the snapshots, as that could be an option considering there were definitely gaps in both systems' audit logs. Thanks, I hadn't thought of that one, I posted on here out of pure frustration when nothing seemed to have caused it and everything looked perfectly fine from what I could see other than the missing business hours lol