Wednesday, June 02, 2021
The point of writing software seems to be writing tests for the software, not in running the software
I've been stuck in testing hell the past few weeks,
what with the
changing
of the
old guard,
and the new guards crew are heavy into the whole “unit test über alles” religion.
I understand the reason behind testing,
and unit testing in particular.
The few times I've tried unit testing,
it has been helpful,
but …
- I think it works best when you start out with unit testing in mind;
- it works best with a module or library;
- you test at the boundary of the module or library and not the implementation;
- you do have tests for your tests, right?
That last bit might sound a bit glib, but it is a real concern. Even Knuth is quoted as saying, “beware of bugs in the above code; I have only proved it correct, not tried it.” I even joked about having to test our tests with both my current (but soon-to-retire) manager and his (soon-to-me-my) manager, because our current regression test has 15,852 individual tests (not all of them might be valid—hard to say, since the tests are automatically generated).
“Project: Lumbergh” was not written with unit tests in mind. Or more specifically, it is a unit. The whole thing. And it's complicated because not only does it implement our business logic (which over ten years has gotten quite complex) but because it has to query multiple databases at the same time across the network (“at the same time” because it has to do its job as a phone call is being made). How does one deterministically test delayed (and/or dropped) queries?
Another issue—“Project: Lumbergh” sends a message to “Project: Cleese” in some circumstances (“Project: Cleese” handles an HTTP notification on behalf of “Project: Lumbergh” because trying to add a TCP connection in a program that deals with UDP would have taken too much engineering time) and we need to check if “Project: Cleese” was notified. I solved that one by mocking the HTTP endpoint that “Project: Cleese” talks to with two interfaces, one for “Project: Cleese” and one for the regression test to query. The issue there is the regression test might ask the mock if it received a request from “Project: Cleese” before “Project: Cleese” gets the notification from “Project: Lumbergh” (a classic race condition). I got it working reliably, but now the regression tests takes over twenty minutes to run, instead of the two-plus minutes (even on the happy path, and I'm not entirely sure why).
It seems like management is more concerned about the tests rather than the product.
