GitHub Shrugs Off Record 1.3Tbps Memcached UDF Reflection DDoS Attack
GitHub experienced sporadic outages over during a 9-minute period. By the 10-minute mark, its systems were fully restored and the attack was successfully mitigated. The DDoS attack was carried out not with an enormous botnet, but with UDP-based memcached traffic.
"Memcached is a tool meant to cache data and reduce strain on heavier data stores, like disk or databases," writes content delivery network (CDN) Akamai. "The protocol allows the server to be queried for information about key value stores and is only intended to be used on systems that are not exposed to the Internet."
Since the memcache protocol doesn't require authentication, it can be abused while UDP traffic is spoofed. Akamai goes on to state that the memcache protocol wasn't meant to be exposed to the internet, however, roughly 50,000 systems around the globe are vulnerable, making them ripe for exploitation by nefarious parties. These individuals used a number of compromised systems to carry out their attack on GitHub.
Interestingly enough, Akamai posted its blog warning about memcached attacks just one day before GitHub got hit.
Luckily for GitHub, an on-call engineer was able to quickly assess the situation (which started at 17:21), and a decision was made in a company chat session to move traffic over to Akamai for additional capacity. "At 17:26 UTC the command was initiated via our ChatOps tooling to withdraw BGP announcements over transit providers and announce AS36459 exclusively over our links to Akamai," wrote Site Reliability Engineering Manager Sam Kottler on the GitHub Engineering blog.
"Routes reconverged in the next few minutes and access control lists mitigated the attack at their border. Monitoring of transit bandwidth levels and load balancer response codes indicated a full recovery at 17:30 UTC. At 17:34 UTC routes to internet exchanges were withdrawn as a follow-up to shift an additional 40Gbps away from our edge."
Kottler went on to add that at no time was customer data compromised, and that this was just a matter of brief inconvenience rather than what could have been a much more devastating problem for the company.
"We understand how much you rely on GitHub and we know the availability of our service is of critical importance to our users," said Kottler. "To note, at no point was the confidentiality or integrity of your data at risk. We are sorry for the impact of this incident."
