Bandwidth.com/interconnect outage affecting US/Canada calls

[SteveITS]

https://status.bandwidth.com/
"Investigating - Bandwidth teams are investigating an incident impacting our V2 Voice services. Customers may notice intermittent service disruptions.
Sep 27, 09:12 EDT"

Per our SIP provider:
"Bandwidth.com a major carrier of VoIP telecommunications is having a major outage across the U.S. and Canada. Most VoIP providers either connect directly through bandwidth or your calls are routed through bandwidth as they interconnect about 40% of the carriers in both countries.
Even if you are not serviced by bandwidth chances are that either your calls are routing through them or the person that you are trying to call is."

I just took one second-hand report from a client who says they were told by one of their customers that has AT&T voice that they were also having problems on some calls, which fits the above.

 

[VasilisV_3CX]

Thank you for reporting @SteveITS.

 

[SteveITS]

For posterity, they were under DDoS attack, and since they connect a lot of calls it affected numerous voice/SIP providers, even calls into VoIP systems from other carriers. We had started getting calls from clients that inbound calls were not connecting, or dropping, or one way audio, or audio would cut off partway through a call. Seems to have cleared up yesterday evening.

 

[Evolute IT]

Many carriers were indeed affected by this.

Fortunately for us, Telnyx wasn't too much affected in our Canadian DIDs and our US DIDs are hosted by another provider which has ported all their inventory from Bandwidth to another backend carrier in emergency, so that too wasn't too affected for now.

Good thing for us, we didn't have anything with VoIP.ms, which is still mostly down from what I understand.

(Please understand the nature of the vagueness since we don't want to help the perpetrators of these attacks.)

 

[ericbumbarger]

Many carriers were indeed affected by this.

Fortunately for us, Telnyx wasn't too much affected in our Canadian DIDs and our US DIDs are hosted by another provider which has ported all their inventory from Bandwidth to another backend carrier in emergency, so that too wasn't too affected for now.

Good thing for us, we didn't have anything with VoIP.ms, which is still mostly down from what I understand.

(Please understand the nature of the vagueness since we don't want to help the perpetrators of these attacks.)

We haven't had much in terms of issues with Telnyx either. We have had a few reports of outbound calls dropping but things have been mostly stable. Seem to be Verizon numbers our customers are trying to call when reporting these issues.

 

[SteveITS]

Verizon numbers

Bandwidth has a Verizon issue today: https://status.bandwidth.com/

"Bandwidth is currently investigating an incident impacting inbound calls from Verizon Wireless to the Bandwidth network. Inbound calls may experience intermittent failures.
Sep 29, 09:46 EDT"

re: emergency porting, I've read it both ways in various places... "we're trying to move them," and "we didn't want to try it since the Bandwidth tool/site was affected and we couldn't reach Bandwidth support."

 

[SteveITS]

It's started up again today, see their status URL I posted.

 

[jcostlow]

One of the carriers we use stated there are more than bandwidth being hit with the DDoS attack but most aren't being named. They rerouted outbound calls this weekend due to the issue and are moving peoples numbers away from bandwidth to have inbound working.

 

[StefanW]

Bandwidth is a core T1 provider in the US and it will have a massive effect as an underlying carrier for many providers you might use. They also provide 911 services in the US which apparently also affected which brought now the Federal gov into play. It goes to show, and am sure Bandwidth had all measures in place, that unsecured IoT devices are the biggest thread in 2021.

In case you are a direct customer off or use a provider using Bandwidth, I would recommend sitting this out and don't port numbers. This takes also days to complete and then what next. For the time been affected switch to email, be open about the current situation (it is not your fault, the IT admin or the provider in my opinion) and advice peer-to-peer communication alternatives for the time been. 3CX offers you many options free of no additional charge. Click2Call, Click2Meet and LiveChat.