Examining the Remnants of a Small DDoS Attack

Setup

I wanted to evaluate Apache's access logs a little closer, so I imported the logs in to an sqlite table. The access logs are in the Combined Log format: %h %l %u %t "%r" %>s %b "%{Referer}" "%{User-agent}" (host ident user time "request" response-code response-size "referer" "user-agent"). In order to turn this in to a tab-delimited file for import in to sqlite, I used the following regular expression:

0

(\d+\.\d+\.\d+\.\d+) (.+) (.+) \[(.+)\] "(.+)" (\d{3}) (\d+) "(.+)" "(.+)"

This regular expression matches each of the relevant pieces of information in to groups that can be used in a replacement expression using your favorite regular expression parser. In my case, I discarded the ident and user fields as they were always empty, further separated the request fields in to three parts (method, request and protocol) and imported all this data in to a table with the following structure:

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

CREATE TABLE "access"
(
    host TEXT NOT NULL,
    time TEXT NOT NULL,
    request_method TEXT NOT NULL,
    request TEXT NOT NULL,
    request_protocol TEXT NOT NULL,
    status_code INTEGER NOT NULL,
    response_size INTEGER NOT NULL,
    referrer TEXT NOT NULL,
    user_agent TEXT NOT NULL
)

Requests

First, taking a look at the total effort:

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

SELECT
    request_method AS method,
    COUNT() AS requests
FROM access
WHERE time >= '2016-11-27 19:54:55'
    AND time <= '2016-11-28 02:56:14'
GROUP BY request_method

method  requests
----------------
GET     55969
POST    9660

The time constraints here represent the first and last requests that were clearly a part of the attack. There was some legitimate traffic in this period but it is far outweighed by the attack traffic. This request reveals that there were 55,969 GET requests and 9,660 POST requests during the 6 hours of the attack. This averages out to about 180 requests per minute, or not very much at all.

But a more fine grained query reveals the timing a little better:

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

SELECT
    strftime('%d', time) AS day,
    strftime('%H', time) AS hour,
    strftime('%M', time) AS `min`,
    COUNT() AS requests
FROM access
WHERE time >= '2016-11-27 19:54:55'
    AND time <= '2016-11-28 02:56:14'
GROUP BY day, hour, `min`
ORDER BY requests DESC

day hour    min requests
------------------------
27  20      41      4230
27  20      39      2550
27  20      35      2448
27  20      38      2418
27  20      40      2346
27  20      37      2232
27  20      34      1980
27  20      44      1872
27  20      45      1872
27  20      36      1836
[...]

Between 20:41 and 20:42 (UTC) the server was actually hit with 4,230 requests (about 70 requests per second). This burst is ultimately what knocked the database server offline by 20:42 (the 15:42 EST noted above).

How much bandwidth did all of these requests chew up?

0
1
2
3
4
5
6
7

SELECT SUM(response_size) as bandwidth
FROM access
WHERE time >= '2016-11-27 19:54:55'
      AND time <= '2016-11-28 02:56:14'

bandwidth
---------
311446609

That's only about 300MB. Consistent with the size of the attack, not very much. It is definitely a lot more than the website in question normally does, but not enough to raise any eyebrows.

Devices

How many devices were involved in this attack?

0
1
2
3
4
5
6
7

SELECT COUNT(DISTINCT host) AS hosts
FROM access
WHERE time >= '2016-11-27 19:54:55'
      AND time <= '2016-11-28 02:56:14'

hosts
-----
311

Apparently only about 300. This is a far cry from the tens or hundreds of thousands of devices believed to be controlled by the news-making Mirai botnet, but apparently more than enough to defeat a $5/mo VPS instance with a non-optimized Drupal 8 installation.

Next up, which hosts did the most leg work?

 0
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

SELECT host, COUNT() as requests
FROM access
WHERE time >= '2016-11-27 19:54:55'
      AND time <= '2016-11-28 02:56:14'
GROUP BY host
ORDER BY requests DESC

host            requests
------------------------
213.251.182.115     3738
212.114.109.218     2784
138.201.151.94      2448
213.251.182.111     2352
213.251.182.105     1992
213.251.182.106     1734
202.67.9.42         1554
184.106.10.128      1368
213.251.182.114     1338
103.45.230.202      1332
[...]

After the top ten, things tend to wind down rather slowly to round out the 311 hosts involved. So what can be found out about these hosts? Not a ton, really, other than who each IP belongs to. There are lots of bulk IP whois services available online, I choose a random one and got the following information:

Most of these devices appear to be hosted on OVH, and Rackspace is another big player represented here. This likely indicates that the devices are either compromised websites or rented machines set up specifically for DDoS'ing. Either way, this attack was not orchestrated from the insecure IoT devices that many fear will continue to grow massive botnets in the coming years.

Locations

There is also a fair distribution of locations for those top ten hosts - France, Netherlands, Germany, Indonesia, the United States and Vietnam are all represented. So where did all of the devices come from?

For a nicer view, open a full screen view of the map above.

This fuller picture reveals that in addition to the high concentration of requests from OVH servers in France, there were also a lot of compromised devices in certain American cities. Who owns the networks these devices sit on? This time around I used InfobyIP's Bulk Lookup Tool, which kindly does not limit the number of lookups per day (or at least I didn't hit the limit):

• In Scottsdale, Arizona, all devices are on GoDaddy.com, LLC (GoDaddy) networks.
• In Brea, California, all devices are on New Dream Network, LLC (Dreamhost) networks.
• In Provo, Utah, all devices are on Unified Layer (United Layer)* networks.
• In Fort Lauderdale, Florida, all devices are on InternetNamesForBusiness.com (INFB) networks.

* The United Layer devices appear to be a small mix of regional web hosting services, with the only major outlier being one device on HostGator.

The total requests in these cities were not nearly as high as what came from OVH, but they helped the total number of devices from the United States more than double those from France. These network owners also further advance the theory that the devices were compromised website. It is possible that they were not able to fire off any many requests as OVH's servers because they are smaller website that hit resource caps.

A lot more digging could be done here to identify and report individual compromised websites, but I suspect that would be an adventure of it's own. And one worth taking if time ever allows...

OOM Killer

Ultimately, the DDoS worked like a charm. It only took about 20 minutes to knock out the database server, completely crippling the Drupal-based website. The burst of 4,000+ requests in one minute caused enough hits against the database to trigger the dreaded oom killer, shutting the service down for good (until I brought it back up manually, anyway).

I have run in to oom killer in other low memory VPS situations in the past and it certainly is not a fun issue to iron out. Luckily, this instance was entirely related to the DDoS attack and not some deeper, more obscure memory issue. Once the attack stopped, things were able to return to normal pretty easily.

In the end, I examined some other logs and decided to restore the machine from a previous image just in case. If the site becomes a frequent target of this sort of attack I will evaluate mitigation options, but I suspect that this particular attack was simply a random test of a small botnet.