Posted by Rameen V. on 03 November 2015 11:49 PM
Current Status @ 17:43 GMT
Server is Online
Network Configuration [Completed]
Server 72 has suffered a catastrophic failure.
We are working on bringing this online as a priority and will be providing updates as progress is made.
Update 01.44AM GMT: We are still working on this. We do not currently have an ETA.
Update 02.30AM GMT: We have not been able to diagnose the hardware fault with the current server. The issue is that the server is experiencing a kernel panic on all kernels we have tried on the server including a livecd. Instead of spending more time in diagnosing this issue further, we're building and deploying a new server and will then begin to restore from our latest backups.
Update 14:06 GMT: We're still working on the restore.. The newly built server exhibited similar issues. We have had to build a different one and are in the middle of performing the restore.
Update 15:44 GMT: The restore is still under way. According to the backup system, it will take around 7-10hrs more for it to complete. If the restore is successful, all websites should be back online shortly after that.
Update 16:47 GMT: Restore progress: 55GB out of 736GB Restored (7.47%)
Update 17:44 GMT: Restore progress: 130GB out of 736GB Restored (17.66%)
Update 21:17 GMT: Due to an error with the restore process, the restore has been temporarily paused. We are continuing to work on this as a top priority. More update to follow.
Update 22:32 GMT: Even though the previous drives were new, they started to exhibit issues which is why the restore was stopped (prematurely by the server).. We've replaced the drives and have started the restore.
Update 17:10 GMT: The server restore has completed. We're currently reconfiguring the network. This is not expected to take very long.
Update 17:44 GMT: We should almost be ready to go. We are just waiting on the on-site techs to switch the network cables around
Update 17:48 GMT: The server is now online. Further maintenance on this server will still be conducted but this should not non service impacting. A full RFO will be emailed to all customers.