Four Lightning Strikes Hit Google’s Belgian Data Center, Minimal Data Lost

We all recognize that backing up data is important, but do you have backups for your backups? A single scheme isn't always enough, and if you think uploading your valuable documents to the cloud is all the protection you need, think again. Some users are finding that out the hard way after one of Google's data centers was affected by four successive lightning strikes.

Google says the incident occurred on August 13, 2015, which is when lightning exacted its wrath on the local utilities grid that powers the company's European data center. There was a brief loss of power to the data center's storage system that's used for what Google dubs the europe-west1-b zone.

Hard Drives
To keep users' data safe, Google destroys all failed drives on site

To be fair, the incident could have been much worse. Google has numerous safeguards in place for these types of things, including automatic auxiliary systems that were able to rapidly restore power. Battery backups are also in place. Nevertheless, some recently written data located on storage systems more susceptible to power failure because of extended or repeated power drain was lost forever.

"In almost all cases the data was successfully committed to stable storage, although manual intervention was required in order to restore the systems to their normal serving state. However, in a very few cases, recent writes were unrecoverable, leading to permanent data loss on the Persistent Disk," Google said.

By "very few," Google is talking about less than 0.000001 percent of Persistent Disk space that suffered permanent data loss.

Water Cooling Tanks
Water cooling tanks sit underneath cooling towers at Google's data center in St. Ghislain, Belgium

"This outage is wholly Google's responsibility," Google added. "However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data."

While this incident was extremely isolated, it underscores the importance for multiple backup schemes for mission critical or otherwise important data that you can't afford to lose.