The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Wednesday, October 05, 2005

I could have been worse

“I've got some bad news,” I said to Smirk.

“You know the rules,” he said. “You have to make it sound like good news.”

I thought for a few moments. “We can save electricity by turning off this server,” I said, pointing to one of the boxes in the office.

“It doesn't work?”

“No,” I said.

“It was hacked?” asked Smirk.

“'Fraid so,” I said.


I only noticed because the network to the office went down. I could ping other boxes in the office, but couldn't get past the firewall. I then went into the data center, logged onto one of the servers and yes, I could still get out to the Internet (relief there). But I couldn't ping the firewall from outside.

I then checked the firewall, and it was still up and running. If I unplug the office network, I could ping the firewall from the outside. Plug the office network in, and it gets swamped.

So something on our network was spamming the network.

Spent the next few minutes plugging and unplugging various bits of the office network until I isolated the culprit—the system that monitors the servers and network.

Of course!

It was at that point that I broke the news to Smirk.


Once it was unplugged, I start poking the machine and yup, there was this odd process: “./s 202.XXX.XXX.XXX”. It was pretty easy to locate the actual executable under /dev/shm/http/, which is normally not a place for executables. The process was running as the apache user, and the files under /dev/shm/http were owned by apache, which to me, is a rather obvious clue that it was Apache was the vector of the exploit.

Among the files in /dev/shm/http/ were configuration files to a hacked IRC bot (which was named s) and some other program (named httpd—not sure what that was) and a few scripts to start things up and to clean the log files (which the cracker had not bothered to run). The hacked up IRC bot would just sit there until commanded to participate in a DDoS attack, which obviously was the cause of the sudden network activity, aimed at a site in Japan.

But how? How did this person get in?

More poking around the system lead to a rather curious request just a few minutes before I lost network connectivity (due to the firewall being swamped)—it was a request to Cacti, a data storage and graphic package with a web based interface written in PHP (why does that not surprise me?). It seems we had fallen prey to a Raxnet Cacti graph_image.php Remote Command Execution Exploit, which basically means the cracker was able to send a command to the server, in this case, a command that would download a perl script and execute it, opening up a shell to a remote connection. It was through this that the hacked programs were uploaded and started.

Fortunately, this is the only system running Cacti, and second, there was no trust mechanism to any other machine on the network from this machine. And third, this is a machine that we only log in to, never log in from, so any damage was limited to just this machine.

On the bright side, it could have been worse.

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: http://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

http://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2020 by Sean Conner. All Rights Reserved.