The point of writing software seems to be writing tests for the software, not in running the software
I've been stuck in testing hell the past few weeks,
what with the
and the new
guards crew are heavy into the whole “unit test über alles” religion.
I understand the reason behind testing,
and unit testing in particular.
The few times I've tried unit testing,
it has been helpful,
- I think it works best when you start out with unit testing in mind;
- it works best with a module or library;
- you test at the boundary of the module or library and not the implementation;
- you do have tests for your tests, right?
That last bit might sound a bit glib, but it is a real concern. Even Knuth is quoted as saying, “beware of bugs in the above code; I have only proved it correct, not tried it.” I even joked about having to test our tests with both my current (but soon-to-retire) manager and his (soon-to-me-my) manager, because our current regression test has 15,852 individual tests (not all of them might be valid—hard to say, since the tests are automatically generated).
“Project: Lumbergh” was not written with unit tests in mind. Or more specifically, it is a unit. The whole thing. And it's complicated because not only does it implement our business logic (which over ten years has gotten quite complex) but because it has to query multiple databases at the same time across the network (“at the same time” because it has to do its job as a phone call is being made). How does one deterministically test delayed (and/or dropped) queries?
Another issue—“Project: Lumbergh” sends a message to “Project: Cleese” in some circumstances (“Project: Cleese” handles an HTTP notification on behalf of “Project: Lumbergh” because trying to add a TCP connection in a program that deals with UDP would have taken too much engineering time) and we need to check if “Project: Cleese” was notified. I solved that one by mocking the HTTP endpoint that “Project: Cleese” talks to with two interfaces, one for “Project: Cleese” and one for the regression test to query. The issue there is the regression test might ask the mock if it received a request from “Project: Cleese” before “Project: Cleese” gets the notification from “Project: Lumbergh” (a classic race condition). I got it working reliably, but now the regression tests takes over twenty minutes to run, instead of the two-plus minutes (even on the happy path, and I'm not entirely sure why).
It seems like management is more concerned about the tests rather than the product.
As much as I wanted the Optimus Maximus, the price of $1,600.00 kept me from buying one, which according to this review was a Good Thing™ (and that Youtube channel is nothing but keyboard reviews—amazing!).
I've been meaning to post this for quite a while, but I've been lax in doing so. Anyway, for my D&D buddies a few videos: one that asks the age old question, “can you go to the toilet in medieval armor?” The second one covers misconceptions about inns, accommodations and taverns in medieval times. These are interesting, but I'm not sure if I'm going to do that level of detail when running my “every-two-week” D&D game.
This last video is mind blowing (at least to me)—fast food in ancient Rome. It's odd to think that “fast food” is as old as Rome—MCROMVLVS anyone? But yes, Roman Empire fast food.
Wlofie, via MeLinkedInstaMyFaceBookWeInGramSpace, sent me to A Mind is Born, a 2½ minute demo program (a program that display some mind-blowing graphics and music) done on the Commodore 64. The impressive thing here isn't that it's done on the C-64, nor the graphics, nor the music, but that he did all that in less than 256 bytes!
For reference, the above paragraph is 351 bytes in size (or 521 if you include the HTML markup). That is inconceivable to me that one can even do something like that with so little.
There is definitely a culture clash at work with regard to testing. I was involved with a four hour discussion about our current regression test with the newer members to the department. The new manager, AG, was saying the regression test should test all the paths in the code. I was saying, yes, that is possible, but many of the new tests would slow the regression test down to a crawl. Then CZ would chime in saying it should be possible to test without slowing down to a crawl. Both AG and I countered that some of the tests involve a timeout, because how would one otherwise test a negative? CZ said we could check the logs, since the logs are logged in a particular order, but I countered that our previous regression test (which runs with the Protocol Stack From Hell) takes over five hours because it checks the logs to ensure we don't miss a log. Using the logs to check to see if something happened isn't deterministic, because the check might come before the action is even logged, and you are still trying to test a negative (in this case, something that should not happen, did not happen, across a distributed, multi-process system) and that slows down the regression test.
We had to leave it with an agreement to disagree on the details for the time being.
Also discussed was a reversion of code in “Project: Lumbergh.” I felt the reversion reverted too much and involved parts that weren't part of “Project: Lumbergh.” CZ felt that was fine, because the changes to the other parts were a part of “Project: Lumbergh” if only indirectly. I countered that in that case, he should have reverted the changes in a different repository that the regression test lives in because that too, indirectly relates to “Project: Lumbergh.” CZ said no, it's in a different repository and thus, shouldn't be reverted. I retorted that the changes he reverted weren't part of “Project: Lumbergh” directly, just as the regression test isn't part of “Project: Lumbergh” directly, and it's only a historical artifact of how our stuff was developed that “Project: Lumbergh,” “Project: Wolowizard,” and “ Project: Clean-Socks” (plus a few others I haven't mentioned) all share a single repository, and that our other repository contains “Project: Sippy-Cup,” “Project: Cleese,” “Project: Seymore” and our current regression test.
We had to leave it with an agreement to disagree on the details for the time being.
Also related to the repositories,
we also had a discussion about the versioning software we use.
I mentioned I prefer
git but we're stuck with SVN for the moment,
with the only thing I like about SVN is the ability to checkout a subdirectory in a repository—it's certainly not because branches are easy,
or SVN tags make any sense to me.
CZ likes SVN for it's ability to easily branch and tags and hates
git's ability to stash files
(we still haven't moved from SVN to
We had to leave it with an agreement to disagree on the details for the time being.
I have a feeling this is going to be a rough few months.
So I wonder—are there unit tests for the various unit test frameworks out there? How are they tested?
Still writing tests. I added checks to see if something that's not supposed to happen didn't happen and I'm only running it over the tests that may actually do the thing that's not supposed to happen (3,456 tests). Doing so adds three seconds of overhead to each test … multiply … carry the one … and hey! Only three hours to run this test run! And oh look! It found something that wasn't supposed to happen did happen!
It's going to be fun reporting at the next meeting that running the full regression test will now take over 13 hours. I'm thinking I'll be tasked with coming up with a different approach to the regression test.
Remember kids! Tests are more important than the program! Testing! Über alles!
Update on Wednesday, June 9th, 2021
Hmmm … it took 259 minutes, 57.5 seconds (4 hours, 20 minutes) to run through the 3,456 tests, so each test took around 4½ seconds, not 3 seconds. Things aren't looking so good for this regression test …
Yesterday, I said the full regression test might take over 13 hours. In light of the results of running just a partial test, it turns out the full regression test will take over 19 hours! The jokes on me though—when I said it would "be fun reporting at the next meeting" I wasn't expecting my new manager to double down on the regression test. Seriously, he asked “Can you run it in parallel?”
So last night I ran a subset of the regression test in 4½ hours and got a few errors where something that shouldn't happen, happened (and it's this “checking for not an event happening” that takes the time). Well, it wasn't a bug in the code being tested, but a bug in the regression test (Surprise! Surprise! Surprise! Only not really). I think that says more about our business logic than it does about CZ or me; both of us attempted to validate this part of the business logic in the regression test, and we both got it wrong.
And about parallelizing the regression test—yes, it's possible. But doing so on the spot isn't. The easy solution is to run the regression test on multiple machines—nice if you have them. The other option is to parallelize the run on a single machine and the code just isn't set up to do that. I'm not saying it's impossible, but it will take engineering effort, and more importantly, testing! Funny how testing your test cases isn't talked about that much.
The slowdown of the regression test is due to “proving a negative”—that is, checking for something that's not supposed to happen did not happen. And in a distributed system like ours, that's not easy to test—a check could happen before the event due to any number of reasons, and how long do you wait to ensure that what shouldn't happen didn't happen?
The other issue to why it will take so long to run is just the sheer number of tests that are run. My “retiring any day now” manager has never been happy with the “shotgun” approach I took to generating the tests—I basically generate thousands of combinations of conditions, most of which “should” never appear in production. But one of those “should never happen” things did happen about seven years ago and well, the less said about that the better. So at least my “shotgun” approach does have the effect of testing for a lot of “I don't know” conditions (most of which are misconfigurations of data from provisioning). And each test we add could potentially double the number of tests cases. I'm sure there's a way to reduce the number of test cases, but to the TDD acolytes out there (and the new management team does appear to follow TDD tenents), “one does not simply reduce the number of test cases.”
And the regression test rolls on …
It's 2:17 am as I'm typing this, sitting on a phone bridge during a deployment of the new “Project: Lumbergh” and I'm glad to know that I'm not the only one with a clicky keyboard. My comment about it brought forth a brief conversation about mechanical key switches, but it was short lived as the production kept rolling out. It's sounding a bit like mission control at NASA. So while I'm waiting until I need to answer a question (happened a few times so far), I thought I might go into some detail about my recent rants about testing.
It's not that I'm against testing, or even writing test cases. I think I'm still coming to grips with the (to me) recent hard push for testing über alles in our department. The code was never set up for unit testing, and for some of the harder tests, like database B returning results before database A, we did manual testing, because it's hard to set such a test up automatically. I mean, who programs an option to delay responses in a database?
It's especially hard because “Project: Lumbergh” maintains a heartbeat (a known query) to ensure the database is still online. Introducing a delay via the network will trip the heartbeat monitor, taking that particular database out of query rotation and thus, defeating the purpose of the test! I did end up writing my own database endpoint (the databases in question talk DNS) and added an option to delay the non-heartbeat queries. But to support automatic testing, I now have to add some way to dynamically tell the mocked database endpoint to delay this query, but not that query. And in keeping with the theme, that's yet more testing, for something that customers will never see!
Then there's the whole “checking to ensure something that shouldn't happen, didn't happen” thing. To me, it feels like proving a negative. How long do we wait until we're sure it didn't happen? Is such activity worth the engineering effort? I suspect the answer from management is “yes” given the push to Test All The Things™, but at times it feels as if the tests themselves are more important than the product.
I'm also skeptical about TDD in general. There's this series of using TDD in writing a sudoku solver:
- OK, Sudoku
- Moving On With Sudoku
- Sudoku: Learning, Thinking and Doing Something About It
- Sudoku 4: Disaster Narrowly Averted
- Sudoku 5: Objects Begin to Emerge
Reading through it, it does appear to be a rather weak attempt at satire of TDD that just ends after five entries. But NO!—this is from Ron Jeffries, one of the founders of Extreme Programming and an original signer of the Manifesto for Agile Software Development. If he gave up on TDD for this example, why is TDD still a thing? In fact, in looking over the Manifesto for Agile Software Development, the first tenent is: Individuals and interactions over processes and tools. But this “testing über alles” appears to be nothing but processes and tools. Am I missing something?
And the deployment goes on …
“99 failing tests in the queue! 99 failing tests! Check one out, grind it out, 98 failing tests in the queue!”
So I'm facing this nearly twenty-hour long regression test and I had this idea—instead of querying the mocked end point if it did its thing or not, have the mocked endpoint check to see if it did its thing or not.
I now send the mocked endpoint the testcase itself
(along with a query flag of
The mocked endpoint will save this information,
and set a
queried flag for this testcase to false.
If it is queried,
it updates the queried flag for the given request.
At the end of the regression test
(and I pause for a few extra seconds to let any pending requests to hopefully finish),
the mocked endpoint will then go through the list of all the testcases it was told about,
and check to see if the query flag and queried flag match—if not, it logs an error.
Sure, now we have two error logs to check, but I think that's better than waiting nearly twenty hours for results.
I got a baseline time for the subset of the regression test without the mock checks—35 seconds. I'm trying to beat a time of 4 hours, 30 minutes with the mock checks.
The new method ran the subset in 40 seconds. The entirety of the regression test, 15,852 tests, took just a few minutes—about the same as before.
I can live with that.
Now all that's left is to write the validation logic—I still don't have it down yet.
The idea of the scrum framework is to organize a development process to move through the different project cycles faster. But does it always incentivize the right behaviours doing so? Many of the users who joined the debate around the question on Stack Overflow have similar stories of how developers take shortcuts, get distracted by their ticket high score, or even feign productivity for managers. How can one avoid these pitfalls?
That the question has been migrated from our workplace exchange to the software engineering one shows that developers consider concerns about scrum and its effectiveness larger than the standard software development lifecycle; they feel its effect on their workplace as a whole. User Qiulang makes a bold claim in their question: Scrum is turning good developers into average ones.
Could that be true?
Ten years later and the prospect of another 24-hour drive to Motown just doesn't sound appealling anymore, my hatred of flying nonwithstanding. So when I was invited to another counsin's wedding a few weeks ago, I decided perhaps it was best to fly this time. And since it has been sixteen years since I last flew, my flying budget was pretty large. It was an easy decision to opt for comformt over price—direct flight, first class, and whatever hoops to speed through Securithy Threater.
And that's how I found myself at 10:30 am sitting at gate D-2 at the Ft. Lauderdale-Hollywood International Airport, watching the line for Starbucks snake clear across the terminal, everybody head down staring at their smart phones, or looking straight ahead talking into the air. Bunny and I were waiting to board the plane for an hour and a half, and that line for Starbucks never got any shorter.
The flight itself was uneventful. I used to fly the Ft. Lauderdale-Detroit flight every year as a kid to spend the summer with my paternal grandparents. Flying back then was way different than today. First off—this was just prior to the airline deregulations, so the various arlines had to compete on service, not price. Everybody was allowed to go to the gate, so family and friends could wait with you there, and family and friends could meet you at the gate on the other side. I do think there was some security—at least a metal detector, but nothing like the security theater of today. But it wasn't cheap—the cost of flying my 10 year old self, adjusted for inflation, is what I paid for a first class ticket to fly myself this year.
I don't think I want to know the price of a first class ticket back in the day.
The service on the flight? It was okay. I mean, a snack box of cheese, crackers and some gummi bears pales to actual meals I remember eating, but the entertainment center in the seat? A wide selection of movies, music, and live flight tracking? That would have blown my 10-year old mine back in the day.
On the down side, the pre-flight safety dance was not performed by the crew, but instead shown on the said entertainment center, after two mandatory airline commercials. Seriously.
I also found it amusing that airplanes still have no-smoking indicators. How long has it been since one could smoke on a plane? Thity years? Fourty years? I wonder at what point they'll finally be removed.
Another change I noticed—now that each seat has a built-in entertainment center, there's no longer an in-flight magazine or Sharper Image catalog.
It seems I only have these odd encounters when on vacation. I was walking towards the lobby of the hotel as half a dozen people, probably early 20s, maybe, were walking the other way. The one in front stops, and asks me directly, “What's your first name?”
I notice he's wearing a name tag attached to a lanyard, and in large bold letters is his name: XXX.
“Sean,” I say. “I see your name is—”
“And what's your last name?” he blurts out before I can finish.
“Cool! What's your middle name?”
“Thanks.” And the group starts walking down the hall.
It was then I noticed the familiar yellow of a school bus through the front door of the lobby.
Bunny and I were at a drug store when I came across this section of empty shelves:
I have no idea what “SquishMallows” are, but whatever they are, stores in the area are unable to keep them on the shevles, even with draconian restrictions. Man, the Barbarians in this area! Keeping the SquishMallows off the shelves like that!
Bunny and I met with my aunts Kay and Jan at Plymouth Roc, a restaurant partly owned by Kay's son Jordan, who is also the groom in the upcoming wedding on Friday.
As we were leaving, Jordan came from the kitchen, bearing a bowl of candied walnuts. “Here you go,” he said. “I made it from my mother's recipe.”
“I never made this!” said Kay.
“But my customers don't know that,” Jordan said.
I don't recall the last time I ate at the Mexican Village, but it's been something like 30 to 35 years. It is my second favorite place to eat, Buddy's being my first. I missed eating at Mexican Village ten years ago, and I wasn't about to miss it this trip. So Bunny and I drove the 25 miles between there and the hotel.
I will say that one thing I do not miss are the Detroit roads—it's like one extended pot hole half-repaired. Ka-thunk ka-thunk ka-thunk bud-a-bud-a-bud-a-bud-a-thunk ka-thunk. The entire way! And back again!
Anyway, other than the road conditions, the trip was mostly uneventful. Things have changed in the 30+ years since I last visited Mexican Village, and let's just say the junction between I-75, I-96, and the Ambassador Bridge is a snarled mess under construction and half-blocked off. And the neighborhood there is … um … shall we say, economically challenged and slowly being reclaimed by nature. It's an odd mix to say the least.
But one oddly maneuvered U-turn in an otherwise abandoned lot later, we arrived. And the place hasn't changed at all in the past 30+ years. I even recogized the same art work on the walls. Amazing!
Way too much food. Just way too much food.
A few hours later, and we were off to an after-rehearsal dinner at The Honey Hole in Plymouth, Michigan. We had the entire restuarant for the party as the groom-to-be, my cousin Jordan, was also a part owner of the place (and is still in the process of reopening the place now that Michigan is slowly opening up). His brother Aaron, owner of the Westwind Brewery in Elkhart, Indiana, was curious about the brewing possibilities in the basement (it's Michigan—every building has a basement, and it's something I sorely miss, living in South Florida), When we were given a tour of the basement, it was clear that brewing beer was not an option, unless you hired a bunch of Hobbits (or Oompa-Loompas—they would work just as well).
And then the food arrived. Way too much food. If this keeps up, I might have to use a baby care bag.
The Honey Hole building is coverned in murals. On one side of the building is this bit of mural:
And on the other side of that wall (inside the restaurant) is this mural:
It's a neat concept, but it cuts a bit too close to home for me.
What can I say about the wedding ceremony? The bride was beautiful. The groom was hansome. The ceremony moving Vows and rings exchanged. Kisses. Crying. Cheering. Photos. It was great!
Okay, I can say a bit more here—the weather forecast for this week was:
- Tuesday: sunny
- Wednesday: sunny
- Thrusday: sunny
- Friday (today): severe thunderstorms and heavy rain
- Saturday: sunny
Today's forecast of severe thunderstorms and heavy rain was a bit concerning, given that the wedding was held outdoors. While it did rain last night and a bit this morning, by the time of the wedding, it was party cloudy and clearing up. By the time the event was winding down at the Fox Hills Summerhouse Outdoor Venue it was clear skys as far as one could see.
Officiating the ceremony was Jordan's brother Aaron. Yes, the same Aaron who owns a brewery. If it seems odd for a religeous ceremony to be delivered by a beer brewer, think again (although Aaron isn't a monk, he can officate a wedding ceremony).
Not much else to say about today, other than good luck to the bride and groom.
The other day Bunny and I were strolling through Northville when I saw this marquee:
It appears that one can actually rent the marquee and have your message displayed to all in Northville. I wonder how much it would cost to say “MY HOVERCRAFT IS FULL OF EELS” and how many people would get the reference?
“Tim Hortons? I thought that was a Canadian chain!”
“It is, but the border to Canada is just over there, to the south.”
“South? Isn't Canada north of the US?”
“Mostly, except for Detroit, which darts out over Winsor, Canada.”
“You know, these donuts don't live up to the hype. They're just okay.”
“I think I prefer the donuts from Publix.”
“Don't let them hear you say that—we may not get out of Michigan alive.”
You better believe they ship! So there's no excuse not to try this pizza.
And no, they did not pay me to say this. I just think it's that good!
My original plan to drive to Buddy's was to take I-275 south to I-96 east to the Davison Freeway, leaving us two blocks south of Buddy's. Then I saw a two mile stretch that wasn't a freeway and had second thoughts (this is Detroit, after all). The backup plan was to then drive I-275 north to I-696 east to I-75 south to the Davison Freeway, but a chance conversation with the front desk revealed that the I-696/I-75 interchange was closed off until I think next year. That meant I went with my original driving plan. I only bring this up, because on the way back to the hotel I caught sight of the “Shrine of the Black Madonna Cultural Center and Book Store / Art Gallery” (the street view from Google). Despite the sign being so fantastic, this is for real and I spent some time going down a historical rabbit hole.
Not only had I never heard of this organization, but its history is mixed with that of the Detroit Riots of 1967, which I had heard of from my grandparents. I also learned that Gordon Lightfoot's song “Black Day in July” was about the Detroit Riots, and the song was banned in 30 states (although I do recall hearing the song as a kid—either because it wasn't banned in Michigan, or my parents had the record, but I do distictly remember that song).
And to think that a chance glance of an odd sign on a building could lead to such an interesting rabbit hole, only because my preferred route to a Detroit restaurant was under construction.
It's our last full day here in Michigan and because we're a) on vacation, and b) lazy, we got a late start on the day (“breakfast” was around 3:00 pm). Around 8:00 pm, Bunny got an idea—we should get some sub sandwiches and store them in the room fridge for when we eventually get hungry. There's a Meijer across the street, and from the web site it appeared they had a deli section. I mean, if Publix sells subs, then Meijer might sell them.
Driving across the street (we drove, in case they did not, in fact, sell subs, we could try elsewhere) was easier said than done. And while the Meijer was still open, the deli counter, sadly, was not. Plan B—Subway. A quick Google search revealed half a dozen Subway shops within five miles, all but one still open.
IT'S STILL DAYLIGHT OUTSIDE!
Buy hey, it was only a few miles away.
And it was closed!
Did I mention it was still daylight outside? Close to 9:00 pm by now, and the sun is still shining in the sky. Bunny noticed a Kroger was in the shopping center, and they were open!
But alas, their deli section was closed.
What is with this town? IT'S STILL DAYLIGHT OUTSIDE!
Yes, we did get some food from a nearby Arby's, but still, IT'S STILL DAYLIGHT OUTSIDE!
You can probably pick this place up for a song:
It's a bit of a fixer-upper, but the lots on either side are completely empty, so you won't have to contend with noisy neighbors.
Of course, there's quite a bit of that in Detroit—a house in need of some DIY TLC in a sea of empty lots.
Notes on a public announcement at the McNamara Terminal of the Detroit Metropolitan Wayne County Airport
“Will Mu Wang please report to gate A-48. Mu Wang, please report to gate A-48. You left your ID there. Mu Wang, please report to gate A-48. Or, you know, Wang Mu. You left your ID at gate A-48.”
The differences between the Ft. Lauderdale-Hollywood International Airport and the Detroit Metropolitan Wayne County Airport are night and day. The Ft. Lauderdale airport is U-shaped; the Detroit airport is two linear terminals, each nearly a mile in length. Hopping through Security Theater was trivial in Ft. Lauderale; it was a nightmare at Detroit. I mean, if you don't want people to use the plastic bins for the contents of pockets, then why are they even there? Stop berating people who use them! The gates at Ft. Lauderdale were loud and crowded; in Detroit—blissful silence. And no Starbucks at the gate.
At least Detroit provides an elevated tramway to traverse the nearly mile-long terminal. That was fun to ride.
The flight back was uneventful the entire way, until we got to the gate at Ft. Lauderdale. The jetway malfunctioned, and we were stuck on the airplane. Everybody was getting up to retrieve overhead luggage and then it was announced that we all had to sit back down because they were planning on taxing the plane to a different gate. So everybody sat down, several minutes of non-movement passed, then suddenly, the jetway moved into place (or someone, or someones shoved it into place) and we could disembark.
And that was really the only issue we encountered. At least it happened on the ground and did not involve the airplane.
Also, it appears the line at the Starbucks had finally cleared. That's nice—took only a week.
Okay, who forgot to tell Bunny and me about the underground tunnel with a color-changing LED light show synchronized to music at the Detroit Metropolitan Wayne County Airport? We had over an hour to kill!
When Bunny and I arrived in Detroit we had our choice of two mid-sized cars to rent—one a Kia, and one a Chevy. Given that we'd be driving in and around Detroit the only safe choice was the Chevy.
It wasn't a bad car. I already drive a Chevy Impala, so the controls were more or less the same. But unlike my car, our rental car had WiFi! I never tried it out, but it was odd to thing that a car could also be a WiFi hotsopt.
Also, this is the first car I've driven that had a push-button start. There was no key—or rather, the only thing we had was a fob, and I never had to take it out of my pocket to unlock the car. Just having the fob near the car, and hitting a button on the door handle was enough to do that. The only times I had to remove the fob from m pocket was to lock the car, and to unlock the trunk.
The only issue I had with the car was the steering wheel—it was a bit too low for me, and I had issues with getting into and out of the car. I had to scrunch up my right leg every time.
And you know what?
It makes sense that there would be a number of computers on a modern airplane to handle the avionics, but I was still surprised at the size of the avionics compartment on an Airbus 350—it's like a mini-datacenter below the cockpit! There appears to be quite a bit of computing power on that airplane. I wonder if part of that is involved with any on-board entertainment systems?