Monday, September 13, 2004
How I spent my hurricane vacation
Well, that was pleasant.
First off, around the 24th of August, the server that
was hosting this site, swift.conman.org, was
compromised (through my login no less). Due to differences in opinion on
the situation (and seeing how the system does belong to Mark) he was kind
enough to give me until Sunday, September 5th
(nevermind a Class 3 hurricane coming through. Then again, Mark will be the
first to admit he isn't the most diplomatic of people).
So when we weren't preparing for Hurricane Frances, moving sites to a different server occupied my time. Then, once Hurricane Frances blew past, the server I moved the sites to got cracked.
It had nothing to do with the swift.conman.org compromise
since we have a very good idea as to who did it. And the major difference
between this compromise and the compromise to swift.conman.org
was the cracker deleting every site on the box, and making sure the box
had to be rebuilt from scratch (since with /bin,
/sbin and the kernel deleted as well, the server wouldn't be of
much use).
So while the National Hurricane Center was bouncing Ivan off of Florida, every moment I wasn't sleeping (or severely sleep deprived) was spent rebuilding three servers (the full extent of the damage) from scratch, and attempting to recover lost data.
Almost makes me pine for the days of getting denial of service attacks.
So once that was done, it meant copying my sites off of
swift.conman.org yet again. Which explains why I haven't been
around all that much during the past two weeks.
It's been a rather interesting two weeks.
And Hurricane Ivan has best be the last storm this year.
Tuesday, September 14, 2004
Some clarifications
I should probably clarify a few things about the hacked servers.
On (or about) August 24th, my shell account was compromised.
This was most likely due to using a compromised Windows system (wth a
keyboard logger) or a Trojaned version of puTTY.exe (an
ssh program freely available for Windows). Not much you can do
except attempt to minimize the damage. Mark and I do have differences of
opinion on how to handle cracking attacks (I tend to be optimistic about
such things; Mark isn't) which caused most of the problems we've had (and
still have, by the way). Since the server was Mark's he felt it best for
everybody on the server to move their sites elsewhere and take the server
down (I now suspect it'll never go back up).
I found no evidance that the machine had been compromised, but Mark thought otherwise. So I moved my sites off to one of the servers I administrate (the ones I had problems with Russian hackers doing denial of service attacks against).
A bit of background on this set of servers. I was hired to administrate four servers—two in Boca Raton (the same facility as Mark's server) and two down in Miami (at the Nap of the Americas). One of the Boca servers had hardware problems so it was decomissioned. Over the past few months I've backed up the sites across each server so that if one goes down, the remaining ones can take over (not automatically, but easily enough). Durring Hurricane Frances' advance towards us, one of the Miami servers crashed. The decision was made to leave it down there until after Hurricane Frances and have the other Miami server pick up the slack (easy enough to do). At the time we weren't certain why the machine crashed, but it did (later on, it was theorized that it crashed during a “test run” of taking the machine down).
The server I moved my sites to was the other Miami server, as I felt that stood a better chance of weathering Hurricane Frances.
On September 8th, the Boca server was compromised.
I honestly feel that the Boca server compromised had nothing to do with Mark's server being compromised. All the websites on the Boca server were deleted, and everything pointed to a single page, giving a shout out to a known person that worked with (or for) the company who had the majority of sites on the Boca server. Also, the Boca server had a certain class of sites on them, one where the updating of the sites was under less control than previously realized (at least by me). And given some evidence (found later on one of the other servers) it appears that the cracker in question had the actual log in information for some of the sites (about half a dozen, and none of them my account) so it points to some form of inside job (again, not much you can do in that case, other than preventing other sites from being wiped out, but this was all found out after the case).
Things were still in place from our preparations for Hurricane Frances (to switch the sites to one or the other server in case of power loss) so I simply enabled the deleted websites on the Miami server, and went in to the Boca facility to retrieve the now dead server. It was during this time that the Miami server was compromised and all the sites (every last site) were deleted.
Later on, I found out that the attacks were timed for the start of the NFL season which is important since the company who has the majority of sites is a gambling/gaming company and the start of the NFL season is an important time of year.
Now, can I say for sure that the compromise of Mark's server was unrelated to the compromised of the other servers? No. Not 100%. Is it likely they're unrelated? Yes. At least in my opinion.
But in the meantime, the servers have been reconfigured and partitioned off with the hope that such an attack will have a less chance of success. The number of accounts has been drastically reduced and of the accounts remaining, the passwords have been changed. The servers are now running the latest version of everything. Will these servers be compromised again? There's always a chance. But hopefully, with some of the changes put in, the damage will be severely limited in scope.
I'm optimistic about that.
