Google Data Center Loses Data After Lightning Strikes

One of Google's data centers located in Belgium has lost some data after being pummeled with four lightning strikes. Unfortunately, this is said to have resulted in some Google users permanently losing access to some of their data, though Google has since managed to access some of the discs that were damaged. This affects the Google Computer Engine (GCE) service, but at this time it isn't known which clients lost data as a result, nor the nature of the data that was nuked. Google has reassured users that while serious, the data loss was also very minimal, permanently affecting only 0.000001% of the disk space.

Google reported that the issue began on Thursday, August 13 at 09.25 PST. The next day Google reported that it was still working on restoring the service of some affected discs. By August 16, Google said that under 0.05 percent of the discs were giving failures. Finally, yesterday Google issued a long statement, explaining in more detail what had happened.

As part of its extensive statement on the event, Google said:

Although automatic auxiliary systems restored power quickly, and the storage systems are designed with battery backup, some recently written data was located on storage systems which were more susceptible to power failure from extended or repeated battery drain ... However, in a very few cases, recent writes were unrecoverable, leading to permanent data loss on the Persistent Disk.

Google says that this "outage is wholly" its own responsibility, and the company says it is continuing work on upgrading its hardware so that it is less susceptible to future power failures, and that most of its Persistent Disk storage is already utilizing such hardware. In addition, Google issued a warning for its GCE clients:

GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data.