1/11/2019: RFO: At 3:13 PM ET Internet connectivity was lost at the LAS data center facility causing all devices homed to this geo-node to lose registration and failover to secondary nodes. The effect of the failover from LAS was to overload the NYJ geo-node which, in turn, caused devices homed to NYJ to intermittently lose registration and failover to secondary nodes. The effect of the mass failover to the DAL geo-node was to overload it, resulting in downtime across all three nodes. The MIN geo-node was not affected by this event as it is not used as a secondary node for any of the other nodes. Steps have been taken to never allow this to happen again. All services are now functioning normally.
1/11/2019: Update #1 as of 3:37pm ET: Concurrent re-registrations of devices failing over from the LAS Node caused contention issues on the other nodes causing registration failures in DAL and NYC. LAS node is back up and we expect registrations to normalize.
1/11/2019: We are monitoring significant packet loss at the LAS colocation facility. We are currently investigating and will provide more details as they become available.