The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Wednesday, June 09, 2021

You know, walking into Mordor might be simple after all

Yesterday, I said the full regression test might take over 13 hours. In light of the results of running just a partial test, it turns out the full regression test will take over 19 hours! The jokes on me though—when I said it would "be fun reporting at the next meeting" I wasn't expecting my new manager to double down on the regression test. Seriously, he asked “Can you run it in parallel?”

Words fail.


“The job of a programmer is to produce test cases.”

So last night I ran a subset of the regression test in 4½ hours and got a few errors where something that shouldn't happen, happened (and it's this “checking for not an event happening” that takes the time). Well, it wasn't a bug in the code being tested, but a bug in the regression test (Surprise! Surprise! Surprise! Only not really). I think that says more about our business logic than it does about CZ or me; both of us attempted to validate this part of the business logic in the regression test, and we both got it wrong.


And about parallelizing the regression testyes, it's possible. But doing so on the spot isn't. The easy solution is to run the regression test on multiple machines—nice if you have them. The other option is to parallelize the run on a single machine and the code just isn't set up to do that. I'm not saying it's impossible, but it will take engineering effort, and more importantly, testing! Funny how testing your test cases isn't talked about that much.


The slowdown of the regression test is due to “proving a negative”—that is, checking for something that's not supposed to happen did not happen. And in a distributed system like ours, that's not easy to test—a check could happen before the event due to any number of reasons, and how long do you wait to ensure that what shouldn't happen didn't happen?

The other issue to why it will take so long to run is just the sheer number of tests that are run. My “retiring any day now” manager has never been happy with the “shotgun” approach I took to generating the tests—I basically generate thousands of combinations of conditions, most of which “should” never appear in production. But one of those “should never happen” things did happen about seven years ago and well, the less said about that the better. So at least my “shotgun” approach does have the effect of testing for a lot of “I don't know” conditions (most of which are misconfigurations of data from provisioning). And each test we add could potentially double the number of tests cases. I'm sure there's a way to reduce the number of test cases, but to the TDD acolytes out there (and the new management team does appear to follow TDD tenents), “one does not simply reduce the number of test cases.”

Sigh.

And the regression test rolls on …

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: http://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

http://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2021 by Sean Conner. All Rights Reserved.