Saturday, September 30, 2006
“In practice … ”
One of the services we offer is Internet connectivity and we have a few clients in this very building, and one of said building customers is freaking out because their firewall (which is managed by a third party) is blocking (and reporting as much) traffic from our network and of course such activity is causing problems on their network. I doubt it's our network causing the problem since:
- They are an exclusive Microsoft Windows shop.
- They have about 100 Microsoft Windows desktop computers in their offices across two floors.
- Their network is flat. As in, one logical segment.
- Microsoft Windows, espcially if you have file sharing enabled (aka NetBIOS), tends to be very chatty on a network segment (lots of broadcast packets looking for other machines with which it can talk to).
- It is my understanding that the majority of users on their network use a web-based application.
- This whole mess is behind a single firewall, which is connected to our network.
My diagnosis is that periodically, their firewall gets swampped with broadcast packets (even if the machines send out a broadcast at random intervals, given enough machines and time, you'll get a brief broadcast storm) or with a lot of people making requests with the web-based application they use (a similiar situation).
I was in their office yesterday with the network analyzer, and plugging it into a port on their switch revealed that about 33% of the traffic is broadcast, with peaks up to 80%. On a flat network, each station (and this includes the firewall) receive all the broadcasts.
This is not good.
But, seeing how they've even outsourced their network to a third company (and not the one that makes the firewall) fingers are being pointed everywhere, so that's why I'm in The Office at midnight on a Friday (technically, Saturday) trying to segment off their traffic.
You see, our network is rather flat at the moment. Not as flat as our customers, nor with the number of machines, but still, they're seeing some of our traffic (not a lot mind you—I've seen their logs) but enough to freak them out.
Now, we have managed switches, and it's easy enough to create a VLAN to isolate traffic (I've done so to a limited degree) but while it's easy on an individual switch, it's not something I've done between network switches, although in theory I know how it's done.
In theory you program the ports on either side of the connection that it's a tagged port (or “trunked”—I've seen both terminology used) and that Ethernet packets being sent down this cable are to be tagged with the appropriate VLAN they belong to.
So I spent much of Friday afternoon double checking my network maps and making sure I knew which ports of which switches had which network traffic, and planning on which VLANs go where.
I figured if I'm going to do this for one customer, I might as well go ahead and do it for everything, since the next chance I get to do this might be quite a while.
Physically, they're plugged into a switch, which is then plugged into a switch/router. If they were plugged directly into the switch/router then I would just isolate that port and be done with it, but since there's another swith between them and us, I have to set up the trunking port (or tagged port, or whatever the terminology is).
Three hours later, and it's not working. I can't get the switch and the switch/router to communicate when using VLANs.
Turn off the VLAN sharing and they talk fine.
And I think it has something to do with the terminology problems I'm having. The switch talks about tagged and untagged ports. The switch/router talks about having trunks. I'm not entirely convinced they're the same thing.
So for now, things are status quo.
Update later today
I talked to Smirk, and he suggested I just go ahead and plug that customer directly into the switch/router and isolate their port, and do that on Wednesday when I'm in The Office.
So, why didn't I do the simplist thing that could possibly work?
Well … because the way I originally wanted to be done would technically be better, as it keeps network traffic from straying.
