“In a system of a million parts, if each part malfunctions only one time out of a million, a breakdown is certain.”
So it's Regression Test Time™ again (for “Project: Wolowizard”) at The Ft. Lauderdale Office of the Corporation, only this time, with new, addtional regression tests!
Okay, it's not too bad. It's a rather simple matter to add the cases to a master list of test cases and expand the program that uses this list to generate the data used for the regression test. That was probably about an hour or so of work. Then a minor change to the actual test program to make sure it fires off the messages under the right conditions (two different messages, ten cases, a 100×100 matrix, but easy enough to code).
Then, generate all the data, copy it all out to the four servers required to run the test, get the latest build of all the programs, move them out to the test servers, make sure the configuration files are up to date on all the servers, make sure The Protocol Stack From Hell™ won't puke, and fire up the regression test.
Only to have one component fail each test because it can't communicate with another component.
SM and I spent the next few hours troubleshooting the issue. The two components are on different servers, but they can see each other. Doing a manual query at the command line shows the query going through. But something deep within the bowels (maybe below the cockles, maybe in the sub-cockle area, maybe in the liver, maybe in the kidneys, maybe even in the colon. We don't know …) of “Project: Wolowizard” is munged.