Wednesday, January 31, 2007
National Gorilla Suit Day
![[Have a Happy National Gorilla Suit Day Today!]](/2007/01/31/ngsd07o.gif "ook ook")
Even though there's no line, there's still a line
All I needed was a single 39¢ stamp.
So on the way to work, I decided to stop by the Post Office because I know they have a machine that sells single 39¢ stamps.
Only the machine is out of order.
Sigh.
I check over in the customer area and amazingly enough there is no line! Suspicious that it might be another holiday I approach, and lo', it's not a holiday! And there are two customers already being helped.
This should only take a minute or two, I think to myself as I get into the non-existant line.
The customer at the far end is talking to the clerk, who makes a dash back into the bowels of the Post Office, only to come back, talk some more with the customer, and dash back into the bowels.
The customer closer to me is patiently waiting while the clerk applies special postal tape to the package, then afixes labels to the package, then fills out said labels, then stamps a whole mess of official looking documents, afixes yet more tape and labels while filling out even more official looking paper work.
Meanwhile, the far customer is still talking to the clerk, who keeps dashing back into the bowels of the Post Office.
A line starts to form behind me. The clerk nearest me finishes wrapping a cacoon around the package, and then I see the customer has two more packages.
So much for it taking a minute, I think, about five minutes later.
Ten minutes after I started waiting, the far customer finally berates the clerk over some small detail, then leaves. Now I can finally buy the single 39¢ stamp.
My alarm clock sounded suspiciously like a phone
Ring.
Ring.
Ring.
“Mughuwaha?”
“Sean,” said Smirk. “You awake?”
“Mughuawhaha,” I said.
“Our customer S is complaining about network lag, what's the address of their installation of Cacti?” I gave Smirk the address, and he was able to log in. “Thanks.”
I fall back asleep again.
Ring.
Ring.
Ring.
“Hugmugahwah?”
“Sean!” It was Smirk yet again. “Somethings wrong with their installation of Cacti—there's no data for the past two weeks.” Oh great, I thought. “Can you look into it?”
“Yeah,” I said, resigning myself to the fact that yes, I am getting up early today.
Notes on IPTables
The problem this morning was a
direct cause of my inability to fully grok iptables. I logged
into the customer's firewall (we offer managed firewalls as one of our
services), which was also running an instance of Cacti to help monitor their network. Sure
enough, the SNMP polling script was failing for some obscure PHP
reason.
Poking around the system, I found a few suspicious files, time stamped
two weeks ago, named ping, ping.1 and
ping.txt. Odd, I thought and when I checked the
contents, yup—a script kiddie script, which opens up a connection to a
remote computer.
Sigh.
More poking around, and I find rather quickly the IRC bot program the script kiddie was running (all files owned by the webserver).
Okay. Cacti has some … issues … with security, and it's no surprise that the script kiddie … exploited … these issues, to install their nefarious wares. And the network latency the customer was experiencing was due to excessive IRC traffic.
The major problem I had was how the script kiddie got access to the
webserver in the first place. Due to Cacti's … issues … with security,
I had explicitly blocked access to all network services with
iptables (with the exception of traffic from The Office).
Only, what I thought I did, and what I actually did were
two different things (much like in practice how theory and practice differ).
I spent several fruitless hours (including blocking all traffic
to the firewall itself but not through the firewall, which
made the remote administration … difficult) before buckling down and
really reading up on how packets flow through iptables.
Now, I had set this up to match our office setup. The only real
difference (and it's a major difference) is our Office Firewall doesn't
NAT, but our
customer's firewall does. Oh, that, and we don't run any services
on our firewall. Two, two major differences between our Office and the
customer are our lack of NATing, services, and an understanding of
iptables. Our three major differences between … oh, I'm
digressing.
About an hour and several hand drawn diagrams later, I finally had a
grasp on the flow of packets through iptables:
![[Flow of packets through IPTables]](/2007/01/31/iptable.gif "Flow of packets through IPTables")
I had the filtering rules in the wrong place, along the packet forwarding path (right hand side of the diagram) instead of the local interface input path (bottom half of the diagram). Once I solved that little problem, then I could concentrate on removing the IRCbots and fixing Cacti (I'm guessing the exploit causes Cacti to stop functioning properly—easiest fix was to reinstall Cacti and make sure I had the file permissions correct).
