Tuesday, October 11, 2005
The Better-Half of my Day Yesterday
At least today is turning out to be a much better day than yesterday.
Yesterday, I had the distinct pleasure of learning just how braindead a Riverstone Networks RS 3000 switch/router can be (and that is a post unto itself).
I also had the distinct pleasure of renumbering (assigning new IP addresses) a thousand sites (literally—I ended up renumbering 1,200 websites), a process that didn't go quite as smoothly as I had wished. This mess started about two weeks ago when R (I manage a few servers for him) informed me that his largest customer (who has the aformentioned 1,200 websites) wanted distinct IP addresses for each one. That meant I first had to secure enough IP addresses and get them routed to the server, which took most of the time.
Meanwhile, I was sent a list of sites from the customer, which was smaller (by about oh … 300 or so) than the actual list of sites on the webserver. So now I had to reconsile both my list and the customer list. I ended up with five lists:
- sites that were in both lists
- sites that I need to add
- sites that I had, but the customer didn't list
- sites that had expired (domain registration expired)
- sites that were no longer hosted with us (found via DNS queries)
Then there was the back-and-forth exchange with the customer that boiled down to: keep all sites (even expired and those that no longer point to the server) and give as many sites their own IP address as possible (I ended up giving the first two lists their own IP, with the last three lists sharing a few addresses).
Then last night was the Great Renumbering.
One thousand (plus) sites. Five lists. Making sure I keep a record of which site gets which address. And yes, it was rough. I started at 2:00 am (technically very early this morning) and ended at around 3:30 am, spending the hour and a half running custom Perl scripts, generating DNS zone files and an Apache configuration file. Messed up a few times—mixing up two lists, forgetting to update the serial number in a zone file, simple mistakes like that.
Then at 7:30 am I get a call from the customer. A couple of domains seem to be down. Stumble over to the computer, log into the server, and find out that I forgot to run one of the lists (no wonder the new configuration file seemed a bit small). Oh, and DNS for a few domains was borked (about six zones out of 1,200 had some custom records). I was surprised at how few sites got broken during the process (and given that the customer was confused about what sites where active, it's even more amazing).
But man, am I tirezzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz