Wednesday, Debtember 17, 2003
“We wouldn't want anything ta happen ta da network, now would we?”
Ah, the joys of getting up the ringing of a cell phone. “Sean,” said R, who owns the servers I'm admining, “the site is down.”
“Mwuggua,” I said.
“Please, check it out,” said R.
“Umyeaokay,” I said, rolling out of bed. I make my way to the Computer Room, ping the backup server. It's alive. I log in. I log in. I log in. It finally sinks in that I was able to log in. And the system load is low too. I then try to bring up a webpage.
Nothing.
Doing it by hand, I see that the web server appears to be wedged. I do a
netstat -an and see hundreds of connections in the
SYN_RECV state. Okay, I think as I consume the Elixer
of the Gods—Coca-cola. Lots of sockets bound up. Need to reset the
webserver. The second I restart it, hundreds of SYN_RECV
connections. Looks like a SYN flood.
With some help from Mark, I tweak some network variables:
sysctl -w net.ipv4.tcp_syncookies=1 and sysctl -w
net.ipv4.tcp_max_syn_backlog=2048 and restarting the web server
helped a bit. Mark then had the idea of rejecting the attacking IP addresses with route add -host
<ip-addr> reject which helped even more (with a script to
automatically do that). Then it was a matter of checking to see if there
were too many attacking IPs,
then running the blocking script. Yet another script to automate
that and the site can still be accessed while under attack.
But that still means the site is under attack and all that traffic from hundreds of machines (at least 500, possibly more) is still flowing across the network, causing havoc. And I doubt it's going to get easier any time soon (the company who's sites are being hosted were already extorted last year—this seems to be a different group … they think).
There isn't much that can be done about a DDoS since most of the attacks now a days are done via compromised machines across the Internet (I recorded attacks from machines from Asia, Europe, the Middle East, South and North America) that basically, you have to prepare for a slashdotting if you want to survive a DDoS, and hope that your provider doesn't kick you out for repeated attacks.
Update on Sunday, January 4th, 2004
Decisions
The peristant attack is now affecting the network where the server is located so two decisions were made: one, to shut down the site being attacked, and two, reinstall the two servers back in Miami and have the attacked site being served from there. I had intended to get as much installed and configured before installing the machines back in Miami, but the attack has moved the timetable up a bit.
Good thing I had everything I needed installed and had configured the IP addresses for the machines (and temporarily set up networking for said IP addresses on my home network).
This does mean that I'll have to get up early tomorrow (ick) and that I'll be missing The Return of the King (as Spring and The Kids are planning on seeing it tomorrow).
On the plus side, these will be billable hours.
