Yesterday T-Mobile suffered a pretty serious network outage, and now the carrier has shed a bit more light on exactly what happened.

Neville Ray, T-Mobile’s president of technology, explains that the event that triggered the outage was a leased fiber circuit failure from a third-party provider in the Southeast. That’s the part of the country where we first started seeing complaints of network issues yesterday.

Ray goes on to say that this happens with every network and that T-Mo built redundancy and resiliency to prevent the issue from affecting customers. However, that redundancy failed, too, causing an overload that was then compounded by other factors.

All of these issues caused an IP traffic storm that spread from the Southeast to create “significant capacity issues” across the IMS (IP Multimedia Subsystem) core network that supports VoLTE calls. T-Mobile said in Q1 2020 that 91% of the total voice calls on its network were VoLTE, so its no surprise that an issue involving VoLTE calling would have such a major effect on customers.

Unsurprisingly, T-Mobile has taken steps to prevent this issue from happening again, including adding permanent additional safeguards.

The outage began at around 9:00 am PT/12:00 pm ET yesterday and T-Mo says that the issues were resolved at 10:00 pm PT/1:00 am ET, so this outage knocked out calling and texting most of the day for many customers. Because of this, it’s good to see Ray and T-Mobile be transparent about what caused the outage.

Source: T-Mobile