Solved 3CX Hosted down

[harrysimpson]

Anyone else experiencing downtime with hosted PBX this morning?

 

[Con Gal]

Clients affected yet again. Do we know what's causing it or is there an update?

 

[WhitebarnIT]

https://status.digitalocean.com/incidents/q4b09b022nmh

 

[Agathocles Prodromou]

Hi all,

There is an intermittent networking issue with DigitalOcean that is affecting mostly the UK region and they are working towards a full resolution. We're in touch with them and will provide another update as soon as possible.

The following connectivity issues have been detected:

• 09:50am UTC ~ 20 minutes.
• 11:04am UTC ~ 15 minutes.
• 11:32am UTC ~ 40 minutes
• 12:12pm UTC ~ 8 minutes

Having said that, we can see that currently all instances are reachable, however this may change until a fix is rolled out to all affected datacenters. We apologize for any inconvenience this may caused and appreciate your understanding.

Please refrain from taking any action on your instance via the customer portal during this time such as stopping/restarting it.

 

[wars]

I'm expecting that this is the reason that all of our SBC's have dropping all day.... The management portal seems to be reachable most/all of the time though.

 

[Damian LAC]

From Digital Ocean

Networking in Multiple Regions​

An update has been posted

The fix has now been rolled out for our SFO2 region and users should be able to interact with their resources in SFO2 normally at this time.

Our team continues to work on the rollout for LON1.

Time posted​

Aug 9, 13:08 UTC

 

[wars]

From Digital Ocean

Networking in Multiple Regions​

An update has been posted

The fix has now been rolled out for our SFO2 region and users should be able to interact with their resources in SFO2 normally at this time.

Our team continues to work on the rollout for LON1.

Time posted​

Aug 9, 13:08 UTC

Just seen this update. Are we SFO2 or LON1?

 

[Agathocles Prodromou]

Just seen this update. Are we SFO2 or LON1?

If you are UK based then your instance is in LON1. SFO2 is in the US.

 

[Damian LAC]

Think UK falls into LON1

 

[policy.stocks.frog]

Another outage for us a few minutes ago

 

[Agathocles Prodromou]

According to DigitalOcean a fix has been rolled out for the LON1 region. On our end we can see all instances being reachable at the moment. We'll be monitoring to ensure everything is stable and provide an update in an hour.

 

[PDenton]

Our clients systems keep going down, can anyone from 3CX update us please?

 

[wars]

Our clients systems keep going down, can anyone from 3CX update us please?

Agreed, still consistently going down despite the fix notification for LON1 which came through 12 mins ago -

 

[VOIPPro]

Yup, still having problems. We have a number of customers reporting poor call quality and drop outs

Latest posting from Digital Ocean: -

Networking in Multiple Regions​

An update has been posted
Our Engineering team saw congestion on network routes between AMS3 and FRA1, which impacted users with services in those regions, as well as traffic on paths between those regions (including services in LON1). Users would have seen latency and connectivity errors.

Although this issue is separate from the root cause of this incident, it was exacerbated by this issue.

Our team has taken action to rebalance traffic and are seeing better performance at this time.

Our team is finishing up the rollout to redundant networking equipment and we're continuing to monitor for any customer impact.

Time posted
Aug 9, 13:52 UTC

 

[Agathocles Prodromou]

Quick update, all instances continue to be reachable and connectivity seems stabilized. An additional routing change has been applied by DigitalOcean to re-balance a network congestion observed on some routes. We will continue monitoring and provide updates until this is confirmed to be fully resolved.

 

[Agathocles Prodromou]

The connectivity of droplets in the UK region remains stable and we have not detected any issue in the past hour. We will continue monitoring.

 

[Agathocles Prodromou]

We haven’t seen any issue with any of the instances and everything seems stable, thus we consider this incident to be resolved. Once again we’re sorry for any inconvenience may caused. Even though such incidents are very rare, rest assured that we will work towards minimizing the possibility of happening again in the future.