The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Wednesday, January 31, 2007

National Gorilla Suit Day

[Have a Happy National Gorilla Suit Day Today!]

Even though there's no line, there's still a line

All I needed was a single 39¢ stamp.

So on the way to work, I decided to stop by the Post Office because I know they have a machine that sells single 39¢ stamps.

Only the machine is out of order.

Sigh.

I check over in the customer area and amazingly enough there is no line! Suspicious that it might be another holiday I approach, and lo', it's not a holiday! And there are two customers already being helped.

This should only take a minute or two, I think to myself as I get into the non-existant line.

The customer at the far end is talking to the clerk, who makes a dash back into the bowels of the Post Office, only to come back, talk some more with the customer, and dash back into the bowels.

The customer closer to me is patiently waiting while the clerk applies special postal tape to the package, then afixes labels to the package, then fills out said labels, then stamps a whole mess of official looking documents, afixes yet more tape and labels while filling out even more official looking paper work.

Meanwhile, the far customer is still talking to the clerk, who keeps dashing back into the bowels of the Post Office.

A line starts to form behind me. The clerk nearest me finishes wrapping a cacoon around the package, and then I see the customer has two more packages.

So much for it taking a minute, I think, about five minutes later.

Ten minutes after I started waiting, the far customer finally berates the clerk over some small detail, then leaves. Now I can finally buy the single 39¢ stamp.


My alarm clock sounded suspiciously like a phone

Ring.

Ring.

Ring.

“Mughuwaha?”

“Sean,” said Smirk. “You awake?”

“Mughuawhaha,” I said.

“Our customer S is complaining about network lag, what's the address of their installation of Cacti?” I gave Smirk the address, and he was able to log in. “Thanks.”

I fall back asleep again.

Ring.

Ring.

Ring.

“Hugmugahwah?”

“Sean!” It was Smirk yet again. “Somethings wrong with their installation of Cacti—there's no data for the past two weeks.” Oh great, I thought. “Can you look into it?”

“Yeah,” I said, resigning myself to the fact that yes, I am getting up early today.


Notes on IPTables

The problem this morning was a direct cause of my inability to fully grok iptables. I logged into the customer's firewall (we offer managed firewalls as one of our services), which was also running an instance of Cacti to help monitor their network. Sure enough, the SNMP polling script was failing for some obscure PHP reason.

Poking around the system, I found a few suspicious files, time stamped two weeks ago, named ping, ping.1 and ping.txt. Odd, I thought and when I checked the contents, yup—a script kiddie script, which opens up a connection to a remote computer.

Sigh.

More poking around, and I find rather quickly the IRC bot program the script kiddie was running (all files owned by the webserver).

Okay. Cacti has some … issues … with security, and it's no surprise that the script kiddie … exploited … these issues, to install their nefarious wares. And the network latency the customer was experiencing was due to excessive IRC traffic.

The major problem I had was how the script kiddie got access to the webserver in the first place. Due to Cacti's … issues … with security, I had explicitly blocked access to all network services with iptables (with the exception of traffic from The Office). Only, what I thought I did, and what I actually did were two different things (much like in practice how theory and practice differ). I spent several fruitless hours (including blocking all traffic to the firewall itself but not through the firewall, which made the remote administration … difficult) before buckling down and really reading up on how packets flow through iptables.

Now, I had set this up to match our office setup. The only real difference (and it's a major difference) is our Office Firewall doesn't NAT, but our customer's firewall does. Oh, that, and we don't run any services on our firewall. Two, two major differences between our Office and the customer are our lack of NATing, services, and an understanding of iptables. Our three major differences between … oh, I'm digressing.

About an hour and several hand drawn diagrams later, I finally had a grasp on the flow of packets through iptables:

[Flow of packets through IPTables]

I had the filtering rules in the wrong place, along the packet forwarding path (right hand side of the diagram) instead of the local interface input path (bottom half of the diagram). Once I solved that little problem, then I could concentrate on removing the IRCbots and fixing Cacti (I'm guessing the exploit causes Cacti to stop functioning properly—easiest fix was to reinstall Cacti and make sure I had the file permissions correct).

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: http://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

http://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2021 by Sean Conner. All Rights Reserved.