Questions to Answer
When an apparent distributed denial of service attack hits your
site, it's easy to panic, especially if it effectively knocks
your site off the Web and the boss is running around like a
beheaded chicken urging you to "do something". This is
even more severe at sites like Fourmilab, where I have to play
both the rôle of the chicken and the beleaguered system
administrator simultaneously. Still, before you can ``do anything''
that's likely to improve the situation, you need to carefully determine
the facts of the matter, which process you can begin by answering the
following questions.
Are we really under attack?
You may have decided your site is under attack because you've observed
a large bulge in the hit rate on your server, seen your inbound
and/or outbound network bandwidth hit the peg, or watched the CPU
load on your server reach saturation.
An attack on Fourmilab.
|
The first question to ask yourself is whether these symptoms are
actually the result of a DDoS attack or something else
entirely. Certainly, all of these things are characteristic
of a DDoS attack, but they do not, by themselves, indicate an
attack is actually underway. The chart at the right shows the onset
of the
DDoS
attack against Fourmilab
in January 2004. Prior to
the attack, the hits per day had been at the typical level of
500,000 to 650,000 per day. As the attack began, hits per day
increased to over a million per day and stayed at that level.
Analysis of the request log showed the additional hits were
completely stylised requests for nothing but the home page,
originating from thousands of different IP addresses all over
the world, with each requestor repeating at a more or less
constant rate, and several hundred new IP addresses "recruited"
into the attacking population each hour. The attack requests were
easily distinguished from legitimate requests for the home page
because they contained request header variables which no browser
was known to send. In this case, an unexpected bulge in request rate
did indeed indicate an attack. But this is not always the case.
Consider the access graph at the left, covering a week in April 2003. Once
again, requests were running at the typical rate with the usual slowdown
during the week-end when, on the 13th of April, the hit rate exploded
to more than double the usual average rate.
Not an attack.
|
This hit the outbound bandwidth hard--its 2 Mb/sec capacity, which is about twice
the average bandwidth requirement, saturated, and response time went from the
typical 100-500 milliseconds to in excess of five seconds for many
requests. The hits just kept on coming at this rate for the first hours
of the 14th as well, at which time they finally began to taper off. An
attack? Nope. Examination of the request log immediately revealed what
was going on. One of the documents on our site had been mentioned on one
of those sites frequented by fat people who cannot spell, who click like
Skinner-pigeons on any link posted there, in order to download, but of course
never read, the contents before posting their erudite "arguements" against
its (unread) content and disdain for the author of its (unread) words.
You can usually identify the source of this kind of traffic bulge from
the "
Referer" field in your HTTP log. (If you don't use the
"combined" log file format which includes this field along with the
"
User-Agent" which identifies the browser, here is an excellent
reason for starting to do so.) In this case, most of the initial hits
to the document in question has a
Referer pointing to a posting
on a site for iconoclasts who all think alike, and a visit there confirmed
that as the source of the hits. One could then breathe a sigh of
relief, "this shall soon pass", and indeed, once the posting scrolled
off the home page of the site, the hit rate returned to normal. A few
minutes of research put an end to the panic over a potential attack
and pointed to patience as the proper prescription.
One month earlier, in March 2003, after a typical week and
week-end, the request rate started climbing from its typical rate
to higher and higher levels on each successive day, reaching
An attack, but not on Fourmilab.
|
more than a million hits a day by the 21st. These hits originated
from all over the world, and were primarily requests for images
from
Earth and Moon Viewer
which, requiring substantial CPU time on the server, vastly exceeded
the server's CPU capacity, sending its load average, which typically
peaks around 3.5 to 4, to more than 400 processes waiting to
run. The referer field in the first hit of these requests were
mostly blank or from search engines. Was Fourmilab under attack?
No--
Iraq was! The outbreak of hostilities in Iraq causes
tens of thousands of people all over the world, having seen satellite
images of Iraq on television, to run to their computers to look
for a source, whereupon Earth and Moon Viewer popped out at the
top of the search. I suspect most of these visitors didn't realise
that the images they were requesting were generated from a
static imagery database, so no matter how far they zoomed
in, they wouldn't be able see the tanks roll toward Baghdad.
What is the kind of attack?
DDoS Attacks of the First Kind: Nuisance Level
DDoS Attacks of the Second Kind: Outbound Bandwidth / Server Saturation
DDoS Attacks of the Third Kind: Inbound Bandwidth Saturation
Where is it coming from?
What is the profile of the attacking hosts?
Is the attacking population static or dynamic?
What is the trend of the attack?
What are the attackers sending?
Is our inbound bandwidth being saturated?
Is our outbound bandwidth being saturated?
Is our server being overloaded?
Are the attack requests stereotyped?
Can attack requests be distinguished from legitimate ones?
What is the request trajectory of an attacker?
When did the attack start?
Who were the first identifiable attackers?
Download Gardol: gardol.tar.gz source archive
by John Walker
March 2004
This document is in the public domain.
