Thursday, Debtember 18, 2003
One serious data center
At 9:00 am, C (the admin I'm replacing) arrived at the Facility in the Middle of Nowhere to pick the two servers and me up for the long trip down to the NAP of the Americas in Miami, were the servers are colocated.
The NAP itself is a six story concrete bunker in Miami, taking up nearly a block unto itself. Fake windows adorn the outside in a rather amusing attempt to make the bunker look less imposing I suppose. Not only is it a six story data center, but from what I understand, it also houses a military command post. We parked in a small lot across a street in what looked to be the back of the building. Carrying a server each, C and I headed to a pair of double doors off to one side. We had to buzz the security desk to be let in.
Then a walk down a corridor, featureless except for the expanse of smoked glass along the wall to our left. At the other end, another set of doors and a buzzer. We were then let inside to a foyer where a security guard looked us over. Apparently pleased that we were there on official business we were then allowed to continue towards the elevator.
This wasn't your standard elevator. About eight feet wide, and the doors opened vertically, top half sliding up, botton half sliding down, leading into a deep, tall elevator with plywood siding. The elevator doors were solid on the outside, but a wire mesh inside and we could watch the walls slide by as we rose to the second floor. Once there, we left the elevator, walked across the foyer to the front desk and signed in. Then C was given the key to the cabinet the machines were slotted for, and we were then buzzed into the machine room.
The rest of the floor was pretty much taken up with the machine room (in fact, I suspect floors two through six are pretty much similar in setup). Rows upon rows of caged racks. Huge pipes snaking through the place filled with cables. Blinken lights everywhere. The hum of thousands of machines filled the air.
Impressive.
C led me through a maze of twisty little passages, all alike.
We slid the machines into the rack, plugged them in, and left. Since I had configured the machines at home, I was pretty confident that it would work—the night before I changed the DNS settings on the site being attacked to point to the new IP address, and had a fresh copy of the site on one of the servers.
It wasn't until we were nearly back at the Facility in the Middle of Nowhere did we get a call informing us that the site wasn't coming up. We drove the few miles back to the Facility in the Middle of Nowhere, only to find that I had no Internet connectivity (it had apparently gone out sometime earlier that day, like around 3:00 am). C then drove me to his house (a few miles away) where I was able to log into the server and fix the problem (forgot to configure Apache to listen on that IP address). Once fixed, I then joined C and some other mutual friends for lunch, where I found out that my cable provider had a major cable cut effecting all of Florida.
But other than that slight misconfiguration, the site came up, and the attacks seem to have slacked off.
