On 19/11/2020 the planned migration between Data Centres exceeded the planned maintenance window. Additionally some un-referenced services were impacted due to DNS resolution issues.
Our technical team has analysed the incident and below are the contributing factors identified:
The shutdown process took significantly longer than had been estimated.
Both storage network switches suffered hardware failure in-transit and would not power-on when reconnected. This is an extraordinarily unlikely occurrence. Reconfiguration of replacements delayed restoration of the storage network which is required to power-on services.
All members of the DNS Cluster were shut down concurrently, impacting internal shutdown and startup processes. This led to a failure of customer domain resolution and impact to services which were out of the scope of this migration.
Although we had allowed contingency time, this was entirely consumed by the compounding issues we faced during the migration.
The following remediation actions are underway:
Network switches in this infrastructure will have configuration automatically backed up, to aid in rapid restoration in the event of any future failures.
Geographically diverse name servers will be added to the DNS Cluster to ensure name resolution continues to function in the event of a localised outage.
Documentation has been updated, including reference to shutdown time required for this infrastructure.
Posted Nov 25, 2020 - 18:48 AEDT
Completed
All Virtual Machines have been brought online and this migration is now complete. If you come across any issues, please get in touch with our support team.
Posted Nov 19, 2020 - 10:10 AEDT
Update
We identified an issue with the storage networking which has now been resolved. We are now starting up all VM's. We will provide updates as things progress.
Posted Nov 19, 2020 - 09:55 AEDT
Update
We are continuing working on bringing these servers online as a priority. We will provide updates as things progress.
Posted Nov 19, 2020 - 09:19 AEDT
Update
There has been a slight delay with bringing all servers online. We will provide updates as things progress.
Posted Nov 19, 2020 - 07:31 AEDT
Update
The servers have arrived at Equinix SY5 and are now in the process of being racked/powered on.
Posted Nov 19, 2020 - 05:32 AEDT
Update
We have now shutdown all of the affected devices and have begun moving them to Equinix SY5.
Posted Nov 19, 2020 - 05:03 AEDT
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Nov 19, 2020 - 03:45 AEDT
Scheduled
As part of our ongoing commitment to ensure all of our services are operating and maintained to our standards, Web24 will be migrating your services to the new, state of the art Equinix SY5 Data Centre.
This migration will commence at 03:45am AEST, and is estimated to take up to 4 hours to complete.
Our teams always endeavour to minimise disruption to our customers. We do anticipate that your service will be offline for up to 4 hours during the maintenance period, however our teams will be working hard to get you up and running as soon as possible.
Additionally, the IP address of your service will remain the same - so you will not be required to make any changes.
Posted Nov 05, 2020 - 17:05 AEDT
This scheduled maintenance affected: Web Hosting cPanel, Reseller Hosting cPanel, VMWare VPS, Dedicated Servers, and DNS Cluster.