The Boston Diaries

Tuesday, October 11, 2005

The Better-Half of my Day Yesterday

Gah.

At least today is turning out to be a much better day than yesterday.

Yesterday, I had the distinct pleasure of learning just how braindead a Riverstone Networks RS 3000 switch/router can be (and that is a post unto itself).

I also had the distinct pleasure of renumbering (assigning new IP addresses) a thousand sites (literally—I ended up renumbering 1,200 websites), a process that didn't go quite as smoothly as I had wished. This mess started about two weeks ago when R (I manage a few servers for him) informed me that his largest customer (who has the aformentioned 1,200 websites) wanted distinct IP addresses for each one. That meant I first had to secure enough IP addresses and get them routed to the server, which took most of the time.

Meanwhile, I was sent a list of sites from the customer, which was smaller (by about oh … 300 or so) than the actual list of sites on the webserver. So now I had to reconsile both my list and the customer list. I ended up with five lists:

sites that were in both lists
sites that I need to add
sites that I had, but the customer didn't list
sites that had expired (domain registration expired)
sites that were no longer hosted with us (found via DNS queries)

Then there was the back-and-forth exchange with the customer that boiled down to: keep all sites (even expired and those that no longer point to the server) and give as many sites their own IP address as possible (I ended up giving the first two lists their own IP, with the last three lists sharing a few addresses).

Then last night was the Great Renumbering.

One thousand (plus) sites. Five lists. Making sure I keep a record of which site gets which address. And yes, it was rough. I started at 2:00 am (technically very early this morning) and ended at around 3:30 am, spending the hour and a half running custom Perl scripts, generating DNS zone files and an Apache configuration file. Messed up a few times—mixing up two lists, forgetting to update the serial number in a zone file, simple mistakes like that.

Then at 7:30 am I get a call from the customer. A couple of domains seem to be down. Stumble over to the computer, log into the server, and find out that I forgot to run one of the lists (no wonder the new configuration file seemed a bit small). Oh, and DNS for a few domains was borked (about six zones out of 1,200 had some custom records). I was surprised at how few sites got broken during the process (and given that the customer was confused about what sites where active, it's even more amazing).

But man, am I tirezzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz

Riverstone Networks RS 3000 sucks!

See, yesterday, all I wanted to do was assign an unused port on the Riverstone Networks RS 3000 from one VLAN to another. Actually, two unused ports, but it's something that, for a switch, should be rather uncomplicated.

Now, the Riverstone Networks RS 3000 works much like a Cisco router—you log in and can type commands at the command line prompt, and even change the configuration through the command line. It will even walk you through the command lines, showing you which options are valid at any point in the command line.

It will even allow you to group ports together into a command. For instance, on our Riverstone Networks RS 3000, a default setting for all the ports are set with one command:

port set et.(1-2).(1-16) speed 100mbps duplex full auto-negotiation off

This sets all 16 ports on the two Ethernet modules to a known setting. And if you need to set a port differently, well …

port set et.(1-2).(1-16) speed 100mbps duplex full auto-negotiation off
port set et.2.8 speed 10mbps duplex half auto-negotiation off
port set et.1.3 auto-negotiation on

So you can set a default, then override later on in the configuration.

The same is true for the port descriptions:

port description et.(1-2).(1-16) "UNUSED"
port description et.1.1 "ns1.example.net"
port description et.1.2 "LINK to switch FOO"
port description et.1.3 "ns2.example.net"

Pretty simple so far (but our hero is walking through an empty house, and finds the stairs to the basement—the audience is shouting at the hero not to go down the stairs. Of course he does).

And the creation of a VLAN is also straight forward:

vlan create TheCompanyVLAN1 port-based id 500
vlan create TheCompanyVLAN2 port-based id 501

This associates a human readable name to a numeric ID that VLANs use. Again, straightforward. You can even assign a block of IP addresses to a VLAN quite simply:

interface create ip TheCompanyVLAN1 address-netmask 10.10.10.1/24 vlan TheCompanyVLAN1
interface create ip TheCompanyVLAN2 address-netmask 10.10.20.1/24 vlan TheCompanyVLAN2

This will automagically set up routing between the two VLANs so far thus defined.

Now, here's where things get interesting. You can assign ports to a VLAN simply enough:

vlan add ports et.(1-2).(1-16) to TheCompanyVLAN1

This assigns all the ports on the Riverstone Networks RS 3000 to VLAN TheCompanyVLAN1. So just like the port speeds and descriptions, it so far looks like you can assign a default VLAN to each port, and then override it futher down in the configuation:

vlan add ports et.(1-2).(1-16) to TheCompanyVLAN1
vlan add ports et.1.4 to TheCompanyVLAN2
vlan add ports et.2.4 to TheCompanyVLAN2

One would think that, but one would be horribly wrong! Muahahahahahahahahahahahahahahaha! YOU CAN'T DO THAT! Muahahahahahahahahahaha! (and here the hero has his face eaten off by a zombie—the audience is going “told you so! Told you not to go into that basement!”)

Try to do that, and the Riverstone Networks RS 3000 will complain bitterly about ports et.1.4 and et.2.4 belonging to VLAN TheCompanyVLAN1. You just can't assigned a default VLAN, then reassign it later (unless it's a trunk port, but then a a trunked port can be assigned to multiple VLANs, and I didn't want that). No, you have to assign each port once to a VLAN.

So, one may then attempt:

comment out vlan add ports et.(1-2).(1-16) to TheCompanyVLAN1
vlan add ports et.(1-2).(1-3) to TheCompanyVLAN1
vlan add ports et.(1-2).(5-16) to The CompanyVLAN1
vlan add ports et.1.4 to TheCompanyVLAN2
vlan add ports et.2.4 to TheCompanyVLAN2

BUT YOU'RE SCREWED! The Riverstone Networks RS 3000 will complain about the ports already being part of TheCompanyVLAN1 so it'll ignore the reassignments of the ports to the same VLAN, it'll bitch about ports et.1.4 and et.2.4 being reassigned, but since you commented out the line that assigns all the ports to TheCompanyVLAN1, it'll then nock the ports out of that VLAN (and here, the hero is now a shambling zombie, stalking his girlfriend to eat her face off).

And if you're unlucky, you'll do this on a live network.

And then all the phones on your desk will start ringing off the hook.

And your cell phone will start ringing.

And you're running around, trying to figure out why the network suddenly took a dump when all you're doing to trying to reconfigure two unused ports to a new VLAN.

Did I mention the part about being screwed?

No, apparently, what you have to do is

comment out vlan add ports et.(1-2).(1-16) to TheCompanyVLAN1
vlan add ports et.(1-2).(1-3) to TheCompanyVLAN1
vlan add ports et.(1-2).(5-16) to The CompanyVLAN1
vlan add ports et.1.4 to TheCompanyVLAN1
vlan add ports et.2.4 to TheCompanyVLAN1

And ignore any warnings that the Riverstone Networks RS 3000 may throw at you, and then check to see if all the ports are set correctly:

RS3000BrainDeathRouter# vlan show
VID VLAN Name       Used For                     Ports
--- --------------- ---------------------------- -----------------------
1   DEFAULT         IP,IPX,ATALK,DEC,SNA,IPv6,L2
500 TheCompanyVLAN1 IP,IPX,ATALK,DEC,SNA,IPv6,L2 et.1.(1-16),et.2.(1-16)

Then go back into the configuration and comment out the two ports:

comment out vlan add ports et.1.4 to TheCompanyVLAN1
comment out vlan add ports et.2.4 to TheCompanyVLAN1

And check to see if that took:

RS3000BrainDeathRouter# vlan show
VID VLAN Name       Used For                     Ports
--- --------------- ---------------------------- -----------------------
1   DEFAULT         IP,IPX,ATALK,DEC,SNA,IPv6,L2 et.1.4,et.2.4
500 TheCompanyVLAN1 IP,IPX,ATALK,DEC,SNA,IPv6,L2 et.1.(1-3,5-16),et.2.(1-3,5-16)

Then bounce back into the configuation, and add the two ports to the other VLAN:

vlan add ports et.1.4 to TheCompanyVLAN2
vlan add ports et.2.4 to TheCompanyVLAN2

And if you're lucky:

RS3000BrainDeathRouter# vlan show
VID VLAN Name       Used For                     Ports
--- --------------- ---------------------------- -----------------------
1   DEFAULT         IP,IPX,ATALK,DEC,SNA,IPv6,L2
500 TheCompanyVLAN1 IP,IPX,ATALK,DEC,SNA,IPv6,L2 et.1.(1-3,5-16),et.2.(1-3,5-16)
501 TheCompanyVLAN2 IP,IPX,ATALK,DEC,SNA,IPv6,L2 et.1.4,et.2.4

And if you're not, all the phones on your desk will start ringing off the hook.

And your cell phone will start ringing.

And you're running around, trying to figure out why the network suddenly took a dump when all you're doing to trying to reconfigure two unused ports to a new VLAN (mmmmmmmmmm—brains).

So, the lesson I'm trying to impart here?

Riverstone Networks RS 3000s suck!

But, if you are in the position of having to use them, then you might want to think ill of grouping ports as part of a command and apply each command to each port separately. Then you stand less of a chance of knocking multiple ports off a VLAN.

Oh, and one more thing. Riverstone Networks RS 3000s are not known to boot fast. Oh no. They take their time. They take at least three minutes to fully boot. Three minutes may not seem like a long time, but when all the phones on your desk are ringing and your cell phone is ringing, three minutes is a XXXXXXX ETERNITY! It's even worse if you have to run to the other side of the building to find a computer you can use to log into the Riverstone Networks RS 3000.

Me? Bitter?

Did I mention that the Riverstone Networks RS 3000 sucks?

[Note to Smirk: I'm not saying we should replace the Riverstone Networks RS 3000, since it's there, it runs, and can otherwise handle the traffic we're throwing at it. I'm just venting a bit at the sheer braindeathness of the software running the darned thing.]