Wednesday, Debtember 17, 2003
“We wouldn't want anything ta happen ta da network, now would we?”
Ah, the joys of getting up the ringing of a cell phone. “Sean,” said R, who owns the servers I'm admining, “the site is down.”
“Mwuggua,” I said.
“Please, check it out,” said R.
“Umyeaokay,” I said, rolling out of bed. I make my way to the Computer Room, ping the backup server. It's alive. I log in. I log in. I log in. It finally sinks in that I was able to log in. And the system load is low too. I then try to bring up a webpage.
Nothing.
Doing it by hand, I see that the web server appears to be wedged. I do a
netstat -an and see hundreds of connections in the
SYN_RECV state. Okay, I think as I consume the Elixer
of the Gods—Coca-cola. Lots of sockets bound up. Need to reset the
webserver. The second I restart it, hundreds of SYN_RECV
connections. Looks like a SYN flood.
With some help from Mark, I tweak some network variables:
sysctl -w net.ipv4.tcp_syncookies=1 and sysctl -w
net.ipv4.tcp_max_syn_backlog=2048 and restarting the web server
helped a bit. Mark then had the idea of rejecting the attacking IP addresses with route add -host
<ip-addr> reject which helped even more (with a script to
automatically do that). Then it was a matter of checking to see if there
were too many attacking IPs,
then running the blocking script. Yet another script to automate
that and the site can still be accessed while under attack.
But that still means the site is under attack and all that traffic from hundreds of machines (at least 500, possibly more) is still flowing across the network, causing havoc. And I doubt it's going to get easier any time soon (the company who's sites are being hosted were already extorted last year—this seems to be a different group … they think).
There isn't much that can be done about a DDoS since most of the attacks now a days are done via compromised machines across the Internet (I recorded attacks from machines from Asia, Europe, the Middle East, South and North America) that basically, you have to prepare for a slashdotting if you want to survive a DDoS, and hope that your provider doesn't kick you out for repeated attacks.
