GitLab Learns Hard Lessons After Production Data Accidentally Gets Nuked

Here at HotHardware, we've talked a lot about the importance of keeping good backups. On a personal level, a failed backup would affect just a couple of people (if not just one); but carelessness in the enterprise can can affect a lot more people. Unfortunately, this is a lesson GitLab has found out the hard way.

As its name implies, GitLab is a website designed to help developers manage their open-source projects from start to finish. Users on the site have an expectation that their data is going to be kept safe, but as is proven here, problem can arise, so we can never treat a single source as a de facto backup.

What happened here is that an IT manager working a late night accidentally deleted the wrong folder, and by the time the mistake was recognized, only a few gigabytes of data was able to be saved (of a few hundred). Following this accident, the company took to Twitter to announce that the server was being taken offline, and then followed-up not long after with an admission of what happened.

We accidentally deleted production data and might have to restore from backup. Google Doc with live notes https://t.co/EVRbHzYlk8

— GitLab.com Status (@gitlabstatus) February 1, 2017

This kind of transparency is great to see, and we hope GitLab's admittance inspires future companies to not waste time in coming forward. Unfortunately for GitLab, even though it took precautions to backup its data, multiple restore methods failed. Imagine being the person having to deal with this reality - it's painful.

When all was said and done, it appears that the company didn't end up losing much data at all (if any - it's hard to tell), and as such, the service has been put back online. Throughout the entire restore process, the company left nothing to the imagination, going as far as sending a new tweet out after the restore process went up a few more percentage points.

While GitLab was able to recover, it ultimately got very lucky. But this is another lesson to learn not only about the importance of keeping backups, but making sure those backups work.