The problem this morning was a
direct cause of my inability to fully grok
iptables. I logged
into the customer's firewall (we offer managed firewalls as one of our
services), which was also running an instance of Cacti to help monitor their network. Sure
enough, the SNMP polling script was failing for some obscure PHP
Poking around the system, I found a few suspicious files, time stamped
two weeks ago, named
ping.txt. Odd, I thought and when I checked the
contents, yup—a script kiddie script, which opens up a connection to a
More poking around, and I find rather quickly the IRC bot program the script kiddie was running (all files owned by the webserver).
Okay. Cacti has some … issues … with security, and it's no surprise that the script kiddie … exploited … these issues, to install their nefarious wares. And the network latency the customer was experiencing was due to excessive IRC traffic.
The major problem I had was how the script kiddie got access to the
webserver in the first place. Due to Cacti's … issues … with security,
I had explicitly blocked access to all network services with
iptables (with the exception of traffic from The Office).
Only, what I thought I did, and what I actually did were
two different things (much like in practice how theory and practice differ).
I spent several fruitless hours (including blocking all traffic
to the firewall itself but not through the firewall, which
made the remote administration … difficult) before buckling down and
really reading up on how packets flow through
Now, I had set this up to match our office setup. The only real
difference (and it's a major difference) is our Office Firewall doesn't
NAT, but our
customer's firewall does. Oh, that, and we don't run any services
on our firewall. Two, two major differences between our Office and the
customer are our lack of NATing, services, and an understanding of
iptables. Our three major differences between … oh, I'm
About an hour and several hand drawn diagrams later, I finally had a
grasp on the flow of packets through
I had the filtering rules in the wrong place, along the packet forwarding path (right hand side of the diagram) instead of the local interface input path (bottom half of the diagram). Once I solved that little problem, then I could concentrate on removing the IRCbots and fixing Cacti (I'm guessing the exploit causes Cacti to stop functioning properly—easiest fix was to reinstall Cacti and make sure I had the file permissions correct).