Wednesday, May 02, 2007
Yeah, yeah yeah, I know … if it ain't broke, don't fix it. But we're phasing out that equipment …
I only stopped by The Office to pick up a cross-over network cable. Oh, I thought, I might as well move the wireless connection to the other router while I'm here. It's not as arbitrary as I make it seem—we're in the process of phasing out one of our routers and the wireless shot to the building behind us is one of the last things still on that particular router. I'm also not in The Office as much as I used to be, so this was a perfect opportunity. It should only take a few minutes, I thought, as I started running a new cable …
An hour later and I think I have things back to normal. I had made the switchover, and it didn't work. After troubleshooting it to the depth of my knowledge (mainly, clearing ARP tables), I decided to give a call to R, the guy who actually handles the wireless shots for us. Turns out, he can't get to some of his equipment. So now we're both trying to trouble shoot the issue when I decide to put things back.
On the way to a customer's site, I get a call from Smirk, asking if I can check on a down server. I tell him I can't as I'm on the way to a customer site to do some work (that required the cross-over cable I mentioned earlier). Since P wasn't available at the time, Smirk said he'd drive to The Office to check it out.
An hour later I'm back at The Office for our weekly meeting. I bump into R, who's headed towards the roof to check the wireless connection. I tell him I'll be in a meeting but if anything comes up, give me a call. I then bump into P, who informed me that the server in question was down because the network cable had worked loose.
Weird.
P and I head over to the meeting and not one minute in the Internet is down. Down hard enough for us to call Dan, the Network Engineer to investigate. Then, R calls, saying his equipment was unplugged from the network!
This is getting weirder by the minute. I excuse myself from the meeting and head over to the Data Center. I meet R there, and sure enough, one of the network cables leading to the wireless shot had been unplugged. Plug it back in, and the wireless stuff comes back online. Then Smirk calls; Dan said our network connection to the Internet was down (small note: we share Internet connectivity with another company—our Internet providers come in through a single router, and are then split out to each company; it was this link between the Internet router and our core router that was down).
Turns out, that network cable was loose.
Gremlins!
We have gremlins in our Data Center!
How else to explain three unplugged network cables?
(Oh, and after everything was plugged back in, I was able to move the wireless connection to the other router without problem—sigh).
