Resolved - Partial service outage - IBM Cloud datacenters

Issue detected - IBM Cloud datacenters are experiencing severe packet loss globally. 

Issue Start - Auckland, New Zealand Wed, 10 June 2020 at 10:00 a.m. NZDT
Issue End - Auckland, New Zealand Wed, 10 June 2020 at 12:15 p.m. NZDT

RedShield is currently investigating a critical issue with one of our datacenter providers, IBM Cloud, which is affecting service to clients connecting to customer websites via RedShield Cloud datacenters in Sydney, Melbourne, San Jose, Washington DC and London datacenters. Connectivity is intermittent at best, and periodically down altogether. 

Applications hosted via RedShield's Auckland or Los Angeles datacenters are not impacted. 

Customers whose applications can be automatically failed over to Auckland or Los Angeles, have been failed over to working datacenters. 

IBM are currently investigating and have not yet advised an ETA for resolution. Further updates will be posted as soon as further information is available. 

Event update:

--------------

11:30 NZDT - Our service provider IBM Cloud confirmed a major issue in their network and they are working to resolve the issue, No ETA at this time.

12:15 NZDT - Some services appear to be improving, however no statement from IBM is yet available. RedShield has detected improved connectivity to many systems but we are unaware as to whether this is indicative of a resolution. 

12:40 NZDT - We have now seen 30 minutes of normal traffic across all customer applications. Our monitors do not currently show any degraded service to any customer globally.

13:40 NZT - We are continuing to see normal network availability, and our monitoring systems do not detect any degraded service globally. RedShield is continuing to monitor networks and customer applications closely for any signs of fault recurrence. 

13:57 NZT - IBM has reported that all services have been restored. https://twitter.com/IBMcloud

Analysis Update:

-----------------

Auckland, New Zealand Thu, 11 Jun 2020 at 3:00 p.m. NZST
Greenwich Mean Time, GMT Thu, 11 Jun 2020 at 3:00 a.m. GMT
Sydney, Australia Thu, 11 Jun 2020 at 1:00 p.m. AEST
San Jose, USA Wed, 10 Jun 2020 at 8:00 p.m. PDT
Washington DC, USA Wed, 10 Jun 2020 at 11:00 p.m. EDT

IBM have advised the following:

IBM Cloud - June 9 Outage
Component: Cloud Platform
Location: Europe, Asia Pacific, North America
Start time: 11 Jun 2020, 5:34 AM
Update time: 11 Jun 2020, 5:42 AM
Severity: Major impact

IBM is focused on external network provider issues as the cause of the disruption of IBM Cloud services on Tuesday, June 9. All services have been restored. A detailed root cause analysis is underway. An investigation shows an external network provider flooded the IBM Cloud network with incorrect routing, resulting in severe congestion of traffic and impacting IBM Cloud services and our data centers. Mitigation steps have been taken to prevent a reoccurrence. Root cause analysis has not identified any data loss or cybersecurity issues.