memcached and massive Denial of Service attack (amplification attack), how to fix it…
Two of the largest distributed denial-of-service attacks in the history of the “World Wide Webs” were launched this past week. This particular attack (called a reflection and amplification attack) took advantage of dumb, default configuration settings, that forced victims to deal with massive amounts of malicious traffic.
Both attacks (against GitHub and another prominent entity) took advantage of memcached servers that sit on the public-facing side of the firewall (aka, the Internet). The attacks were detected by Arbor Networks where peak traffic spiked at a whopping 1.7 terabits per second.
GitHub, in its good sense, called for help from Prolexic, who intervened and routed all traffic to its scrubbing data centers, detected and blocked out the malicious packets. Eight minutes later, the assault tapered off. Their smart response made this, the “Mother of all DoS attacks”, an eight minute issue. Impressive. But, the whole attack really shouldn’t have been possible (or rather, shouldn’t have been let to happen).
The Mother of all DoS attacks.
Illustration 1: The record-setting 1.7 Tbps attack
It Takes A Village
The internet community rallied greatly this week to shut down access to many of the open memcached servers – but obviously not all of them. Due to thousands and thousands of servers still running memcached in default-open mode, this will definitely perpetuate the recurrence of other similar attacks for several more days, weeks, or maybe months. The illustration below shows the on-going nature of the attacks.
Illustration 2: Qihoo 360 Technology’s DDoS Mon capture of memcached attack over last 10 days.
What are Memcached Servers Used For
Memcached servers allow applications, that need to access large volumes of data from an external database, to cache some of that data in memory. This allows for better performance by the application versus retrieving data from the database individually. Many thousands of businesses worldwide use memcached servers to improve page loading times and to handle demand-spikes more efficiently. This approach has been commonly used for the last decade
Typically, these servers are used internally, away from the public internet and accessible only from within a trusted-network to improve internal application to db performance. But many lazy or underfunded developers have left memcachecd servers exposed to the world-wide-interwebs, where they can be discovered and exploited by…. um, anyone. Essentially, anyone/everyone can query a memcached server, and they will cordially respond, “welcome, no authentication required”.
To make matters worse, shortly after the attack was discovered, there were three proof of concept exploits introduced into the wild with various lists of 100,000 plus vulnerable memcached servers lurking around the Internet. At the time of this writing, there are now approximately 40 full exploit packages in the wild waiting for some dummass to leverage in an attack. I’ll neither post the exploit code nor the list of vulnerable servers for ethical reasons.
The ultimate problem, besides having an application designed for internal-only use sitting in the public, is that memcached servers have the UDP port configured to “open” as the default.
You can test if your memcached server is vulnerable to amplification attack by running:
###>>> echo -en “\x00\x00\x00\x00\x00\x01\x00\x00stats\r\n” | nc -q1 -u 127.0.0.1 11211
But Why UDP? For Gosh Sakes…
UDP (user datagram protocol) is part of the basic communication protocols within the IP (Internet Protocol). Sound familiar? It should, since IP is part of the Internet’s foundational building blocks: TCP/IP (Transmission Control Protocol/Internet Protocol). Like other protocols within the IP, UDP is relatively dumb in the sense that it doesn’t require handshaking or ACK like the protocols within TCP.
When the Internet was “safe”, UDP was innocently chosen for memcached because it was just-plain faster than TCP. While use of these dumb protocol are rarely used by themselves today, there are still many apps and techniques (beyond memcached servers) that defiantly use UDP out there.
Illustration 3: Memcached servers allow important data to be accessed more quickly by an application than by querying data stored in the database. (MySQL Image)
How to Fix It, Please Fix It!
First, the best way to fix this would be to NOT HAVE memcached servers sitting in the open Internet (e.g., like a proverbial sitting duck).
Shortly after this new attack was revealed, the development team behind the memcached open-source project released a new version that locks down the UDP port. But fixing the issue within the memcached server’s config file is really very easy. Still, memcached servers shouldn’t be sitting “out there”.
Because memcached listens on INADDR_ANY and runs with UDP support ENABLED by default, admins should disable UDP support all together if they are not using it for a specific controlled reason.
Below is a general example of config file fix, which will be similar for most Unix(s).
###>>> sudo vi /etc/sysconfig/memcached
OPTIONS="<span class="highlight">-l 127.0.0.1 -U 0</span>"
<escape>:wq! (save the file and quit vi)
###>>> sudo service memcached restart
The easiest way to prevent your Memcached servers from being abused as reflectors is to firewall, block or rate-limit all UDP on source port 11211. But this port would only deal with memcached, not the other vulnerable apps using other UDP ports, and there are 65,535 UDP ports
Internet Service Providers need to come up with better systems to detect IP spoofing, which is the root of what allows bad-actors to pretend to be someone else to launch their attacks. That’s a whole ‘nother conversation…
The potential threat landscape created by memcached reflection and amplification cannot be easily defended against by ISPs, because IP-spoofing is permitted on the Interwebs.
Here are the details related to the vulnerability
Chinese firm Qihoo 360 Technology provides a passive DDoS and Botnet monitoring called DDosMon. They recent stood up a greatly helpful extension of their service for specifically tracking memcached-based DDoS attacks. Check this to see all the action in REAL-TIME. Qihoo 360 Technology also published an insightful blog post detailing updated statistics related to the attack, the victims, and attack sources.
Oh, And There’s Another Dirty Little Secret
Since memcached was designed to be used without logins or passwords, attackers can also remotely steal any sensitive user data that’s been cached with ZERO (repeat ZERO) authentication. This data may include confidential database records, emails, website customer information, personal data, authentication data, API data, Hadoop information, and more.
Think about the General Data Protection Regulation (GDPR) for a minute. Customer Information breached = Massive Fines.
By using a simple debug command switch, bad-actors can reveal the ‘keys’ to your data and retrieve data from anywhere on the planet. Additionally, it is also possible to maliciously modify and re-insert data into the cache without the memcached owner even knowing.
Server administrators are strongly advised to install the latest Memcached 1.5.6 version which disables UDP protocol by default to prevent amplification/reflection DDoS attacks and disable these debugging features.
In closing, if you use memcached servers, check and correct your configuration files. Restrict UDP in your memcached server config files. Put these things behind firewalls, and update your firewall rules to DENY this traffic.
Hope this helps.
Until next time,
PS: the picture on this blog post is a UDP joke. You have to know how the protocol works to get the joke… Even then, it’s not hilarious. Cheers.