GitHub Shrugs Off Record 1.3Tbps Memcached UDF Reflection DDoS Attack

Code distribution site GitHub was hit with a massive distributed denial-of-service (DDoS) attack yesterday afternoon, but thanks to prior planning and automatic routines to counter such attacks, it was able to come through [relatively] unscathed. At its peak, GitHub was inundated with a record 1.35 Tbps of traffic, and was subsequently hit with another brief 400 Gbps burst of traffic.

GitHub experienced sporadic outages over during a 9-minute period. By the 10-minute mark, its systems were fully restored and the attack was successfully mitigated. The DDoS attack was carried out not with an enormous botnet, but with UDP-based memcached traffic.


"Memcached is a tool meant to cache data and reduce strain on heavier data stores, like disk or databases," writes content delivery network (CDN) Akamai. "The protocol allows the server to be queried for information about key value stores and is only intended to be used on systems that are not exposed to the Internet."

Since the memcache protocol doesn't require authentication, it can be abused while UDP traffic is spoofed. Akamai goes on to state that the memcache protocol wasn't meant to be exposed to the internet, however, roughly 50,000 systems around the globe are vulnerable, making them ripe for exploitation by nefarious parties. These individuals used a number of compromised systems to carry out their attack on GitHub.

Interestingly enough, Akamai posted its blog warning about memcached attacks just one day before GitHub got hit.

github ddos

Luckily for GitHub, an on-call engineer was able to quickly assess the situation (which started at 17:21), and a decision was made in a company chat session to move traffic over to Akamai for additional capacity. "At 17:26 UTC the command was initiated via our ChatOps tooling to withdraw BGP announcements over transit providers and announce AS36459 exclusively over our links to Akamai," wrote Site Reliability Engineering Manager Sam Kottler on the GitHub Engineering blog.

"Routes reconverged in the next few minutes and access control lists mitigated the attack at their border. Monitoring of transit bandwidth levels and load balancer response codes indicated a full recovery at 17:30 UTC. At 17:34 UTC routes to internet exchanges were withdrawn as a follow-up to shift an additional 40Gbps away from our edge."

Kottler went on to add that at no time was customer data compromised, and that this was just a matter of brief inconvenience rather than what could have been a much more devastating problem for the company.

"We understand how much you rely on GitHub and we know the availability of our service is of critical importance to our users," said Kottler. "To note, at no point was the confidentiality or integrity of your data at risk. We are sorry for the impact of this incident."

Tags:  Akamai, DDoS, GitHub, memcache