The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Wednesday, Debtember 17, 2003

“We wouldn't want anything ta happen ta da network, now would we?”

Ah, the joys of getting up the ringing of a cell phone. “Sean,” said R, who owns the servers I'm admining, “the site is down.”

“Mwuggua,” I said.

“Please, check it out,” said R.

“Umyeaokay,” I said, rolling out of bed. I make my way to the Computer Room, ping the backup server. It's alive. I log in. I log in. I log in. It finally sinks in that I was able to log in. And the system load is low too. I then try to bring up a webpage.

Nothing.

Doing it by hand, I see that the web server appears to be wedged. I do a netstat -an and see hundreds of connections in the SYN_RECV state. Okay, I think as I consume the Elixer of the Gods—Coca-cola. Lots of sockets bound up. Need to reset the webserver. The second I restart it, hundreds of SYN_RECV connections. Looks like a SYN flood.

With some help from Mark, I tweak some network variables: sysctl -w net.ipv4.tcp_syncookies=1 and sysctl -w net.ipv4.tcp_max_syn_backlog=2048 and restarting the web server helped a bit. Mark then had the idea of rejecting the attacking IP addresses with route add -host <ip-addr> reject which helped even more (with a script to automatically do that). Then it was a matter of checking to see if there were too many attacking IPs, then running the blocking script. Yet another script to automate that and the site can still be accessed while under attack.

But that still means the site is under attack and all that traffic from hundreds of machines (at least 500, possibly more) is still flowing across the network, causing havoc. And I doubt it's going to get easier any time soon (the company who's sites are being hosted were already extorted last year—this seems to be a different group … they think).

There isn't much that can be done about a DDoS since most of the attacks now a days are done via compromised machines across the Internet (I recorded attacks from machines from Asia, Europe, the Middle East, South and North America) that basically, you have to prepare for a slashdotting if you want to survive a DDoS, and hope that your provider doesn't kick you out for repeated attacks.

Update on Sunday, January 4th, 2004

Why I did what I did during a DDoS attack


Decisions

The peristant attack is now affecting the network where the server is located so two decisions were made: one, to shut down the site being attacked, and two, reinstall the two servers back in Miami and have the attacked site being served from there. I had intended to get as much installed and configured before installing the machines back in Miami, but the attack has moved the timetable up a bit.

Good thing I had everything I needed installed and had configured the IP addresses for the machines (and temporarily set up networking for said IP addresses on my home network).

This does mean that I'll have to get up early tomorrow (ick) and that I'll be missing The Return of the King (as Spring and The Kids are planning on seeing it tomorrow).

On the plus side, these will be billable hours.

Obligatory Picture

[The future's so bright, I gotta wear shades]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: https://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

https://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.