Tuesday, July 03, 2007
The dreaded “Early Morning Call”
Ah, nothing like getting a call at the ungodly hour of 10:30 to fix a munged firewall. For reasons I'm still trying to figure out it apparently crashed last night and the fix was a power cycle.
But in looking over the log files, I kept seeing:
Jul 2 08:48:07 localhost kernel: SPOOF-SRC-172 IN=eth0 OUT=eth1 SRC=172.21.159.36 DST=XXX.XXX.XXX.XXX LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=0 DF PROTO=TCP SPT=11100 DPT=1516 WINDOW=0 RES=0x00 RST URGP=0
I had programmed the firewall to reject any private IP address (since you're not supposed to see private IP addresses on the Internet at large) yet here I was, seeing a private, albeit a less known private, IP address. Only the source address kept changing for each packet logged, but always within the 172.21.0.0/24 block.
This meant that the address was originating somewhere on our network, but where?
Pinging one of the private IP addresses from the router directly connected to the firewall didn't show anything. It seems that the ping packets were sent to the main core router, which just dumps private IP addresses to the bit bucket.
But why was that router throwing anything from 172.21.0.0 to the firewall?
Now, the rule I have for filtering these packets in iptables
looks like:
iptables --table filter --append FORWARD --source 172.16.0.0/12 --jump LOG --log-prefix "SPOOF-SRC-172 " iptables --table filter --append FORWARD --source 172.16.0.0/12 --jump REJECT iptables --table filter --append FORWARD --destination 172.16.0.0/12 --jump LOG --log-prefix "SPOOF-DST-172 " iptables --table filter --append FORWARD --destination 172.16.0.0/12 --jump REJECT
which doesn't record the MAC address (which uniquely identifies an Ethernet interface on a network). However, by adding:
iptables --table mangle --append PREROUTING --source 172.16.0.0/12 --jump LOG --log-prefix "WHAT "
the MAC address is logged.
Cool!
Jul 3 17:48:24 localhost kernel: WHAT IN=eth0 OUT=MAC=00:10:e0:02:09:dd:00:02:85:12:a7:c0:08:00 SRC=172.21.117.49 DST=XXX.XXX.XXX.XXX LEN=40 TOS=0x00 PREC=0x00 TTL=239 ID=0 DF PROTO=TCP SPT=11112 DPT=3716 WINDOW=0 RES=0x00 RST URGP=0
What's recorded is the MAC address of the interface (the first six octets), followed by the MAC address the packet came from (the next six octets) followed by the protocol (the final two octets, and in this case, it's an IP packet).
Now, the first three octets of a MAC address are the vendor code—who makes that particular Ethernet device. Easy enough to look up and—
A Riverstone?
It's the router?
There's nothing on our network that uses the 172.16.0.0/12 private IP space. And yet, here's one of our routers, sprewing forth garbage packets.
Sigh.
Time to get with Dan the Network Engineer on this …
Update on Thursday, July 5th, 2007
Between Dan the Network Engineer and I, the problem was solved.
