Wednesday, July 09, 2008
It's hard to troubleshoot when the gun is jammed …
Smirk gave me a router to configure for a customer and asked that I save the current configuration before I make any changes. It's a Cisco router, so saving the configuration is just copying it to a TFTP server. Nothing terribly hard—I already have a TFTP server installed on my workstaion, and I can easily configure the spare network port on my workstation to match one of the ethernet ports on the router, string a cable (which I already have lying around) and voilà. Nothing to it.
One Hour Later …
I'm ready to kill. I'm thinking one of (or all of) the CentOS maintainers, since I can't imagine the maintainers of logrotate
, syslogd
, xinetd
or tftpd
(is that even maintained any more?) all botching it in a perfect storm of stupidity.
tftp
is controlled and run by xinetd
. But the Cisco router can't save the configuration file. It's not a networking issue since I'm logged into the Cisco router over the network. The configuration in xinetd
to run tftpd
looks good, but what the heck, I decide to restart xinetd
and check the logs anyway. That's when I find that logrogate
isn't. Or rather, it is, but it's failing to tell syslogd
to reopen its log files. And the configuration files for logrotate
look okay as far as I can tell.
But even worse, syslogd
appears to be buffering the logs in memory, maybe updating the files every 30 seconds or so. And the syslogd
configuration I have specifically states otherwise. It could be that xinetd
is really slow, but I have a hard time believing that it takes xinetd
a full twenty seconds to digest a 13 line configuration file (and there are about 30 such files—do the math) on a 2.6GHz dual-core box—the harddrives aren't that slow (heck, Firefox comes up faster than that on this system).
Sigh.
I really hate it when the tools I expect to work don't. It's one thing to know going in that the tools don't work, or aren't there. I know I'm going to get sidetracked and can make better time estimates. It's making a time estimate expecting the tools I work with to work and then getting sidetracked debugging tool issues that pisses me off, because now I'm under a time crunch.
It's even more frustrating when the customer isn't even there to accept the router.