The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Wednesday, March 01, 2023

I'm seriously wondering who is trolling who at this point

I have a Gmail account. I signed up early enough to get my name as an email address at Gmail. But I never use it for anything, so by default, anything that arrives there is either spam or misaddressed. I will occasionally check it, and I found two emails from one Trudy XXXXX­XXXXX­X. The first one:

From
Trudy XXXXX­XXXXX­X <XXXXX­XXXXX­XXXXX­XXXXX­XX> (an address from a Tennessee school)
To
Sean Conner <sean.conner@gmail.com>, (and other addresses to the same Tennesee school, I checked)
Subject
IMPORTANT DEVICE INFORMATION (Waiver)
Date
Mon, 20 Feb 2023 15:51:00 -0500

Parents/Guardians:

Please check your child's Power School account to see if your child has a hold due to a damaged, stolen or lost device. If your child has a damaged, lost, stolen device, you may be eligible to fill out a waiver to allow your child to get a device for no charge.

-The waiver is limited to devices only. Keyboards, cases and chargers are not covered.

If you have already filled out a waiver, or your child's device has been returned to school in good working condition, please disregard this message.

Thursday, February 23, 2023, is our Black History Program. You will be allowed to fill out the waiver after each program (morning or afternoon show).

Part of the waiver agreement requires you to have custody of your child as shown in Power School, provide a valid ID, and be prepared to watch a short 2.5-minute video at the time of completing the waiver.

Morning show is at: 9:15 am
Afternoon show is at: 2:00pm

Thank you 🙂

[School Logo]

Trudy XXXXX­XXXXX­XXX
School Counselor
XXXXX­XXXXX Elementary
XXXXX­XXXXX­XXXXX­XXXXX­XX
Memphis, TN 38116
901-XXXXX­XXX (School)
901-XXXXX­XX (Fax)
XXXXX­XXXXX­XXXXX­XXXXX­XX

"Every child deserves to be a champion: an adult who will never give up on them, who understands the power of connection and insists they become the best they can possibly be"-Rita Pierson

Together, we MUST BELIEVE.

Together, we WILL ACHIEVE.

Together, we ARE REIMAGINING 901!

Does the Sean Conner who lives in Tennesee not know his own Gmail address? I always wonder about that. But regardless, I decided to reply with a bit of surrealism.

From
Sean Conner <sean.conner@gmail.com>
To
Trudy XXXXX­XXXXX­X <XXXXX­XXXXX­XXXXX­XXXXX­XX>
Subject
IMPORTANT DEVICE INFORMATION (Waiver)
Date
Wed, 1 Mar 2023 14:48:00 -0500

One question I have is—what do I do if I don't have a child? The device is fine, but I don't have a kid that uses it. Do I still need a waiver?

The second question I have is—why is my non-existent child enrolled in a school in Tennessee when I live in Florida?

Thank you.

I expected no reply, or maybe a reply like “Sorry to have bothered you.” What I did not expect was this reply:

From
Trudy XXXXX­XXXXX­X <XXXXX­XXXXX­XXXXX­XXXXX­XX>
To
Sean Conner <sean.conner@gmail.com>
Subject
RE: EXTERNAL - Re: IMPORTANT DEVICE INFORMATION (Waiver)
Date
Wed, 1 Mar 2023 14:55:00 -0500

Hello,

If you have device that belongs to XXXXX­X County Schools, we will need to arrange a way to get the device back or you will need to pay for the device.

You cannot get a waiver if your child no longer attends XXXX

Do you know if you are listed as a contact for a relative of step child who attends XXXXX­XXXXX Elementary? If so what are the names?

You will need to contact the parents/ guardians and ask them to remove you from their child’s contact.

I hope that I was able to assist you

Sent from Mail for Windows

What? My email was taken seriously? Am I being trolled? Who is trolling who?

Wow.

So anyway, the other email I received from Trudy (I mean, aside from the reply I received for the first email) was this one about current attentance policies:

From
Trudy XXXXX­XXXXX­X <XXXXX­XXXXX­XXXXX­XXXXX­XX>
To
Sean Conner <sean.conner@gmail.com>, (and other addresses to the same Tennesee school, I checked)
Subject
School Wide Attendance/Chronic Absence IMPORTANT
Date
Mon, 20 Feb 2023 17:42:00 -0500

Greetings XXXXX­XXXXX Family!

Last week we had two great days of overall school attendance, however two days is just not enough. Please make sure that you are sending your child to school every day and on time. Please review the attendance policy about excused absences. Moving forward, we will adhere strictly to the attendance policy when excusing absences.

Attendance and Excuses (Policy #6014) The XXXXX­X County Board of Education believes that regular attendance is a necessary requirement of all students. All students are expected to attend school on each day that school is officially in session and remain at school for the entirety of the school day. Only the following reasons will be considered for excused absences:

  1. Illness, injury, pregnancy, homebound circumstance, or hospitalization of student. The District may require a parent conference and/or physician verification to justify absences after the accumulation of ten (10) days of absence during a school year. Notes must be date specific and will be required for subsequent absences beyond ten (10) days.
  2. Death or serious illness within the student's immediate family.
  3. When the student is officially representing the school in a school sponsored activity or attendance at school-endorsed activities and verified college visits.
  4. Special and recognized religious holidays regularly observed by persons of their faith. Any student who misses a class or day of school because of the observance of a day set aside as sacred by a recognized religious denomination of which the student is a member or adherent, where such religion calls for special observances of such day, shall have the absence from that school day or class excused and shall be entitled to make up any school work missed without the imposition of any penalty because of the absence.
  5. A court order; a subpoena; and/or a legal court summons.
  6. Extenuating circumstances over which the student has no control as approved by the principal.
  7. If a student's parent, custodian or other person with legal custody or control of the student is a member of the United States Armed Forces, including a member of a state National Guard or a Reserve component called to federal active duty, the student's Principal shall give the student: a. An excused absence for one (1) day when the student's parent, custodian or other person with legal custody or control of the student is deployed; b. An additional excused absence for one (1) day when the student's parent, custodian or other person with legal custody or control of the student returns from deployment; and c. Excused absences for up to ten (10) days for visitation when the student's parent, custodian or other person with legal custody or control of the student is granted rest and recuperation leave and is stationed out of the country. d. Excused absences for up to ten (10) days cumulatively within the school year for visitation during the deployment cycle of the student's parent, custodian or other person with legal custody or control of the student. Total excused absences under this section (c) and (d) shall not exceed a total of ten (10) days within the school year. The student shall provide documentation to the school as proof of the deployment of the student's parent, custodian or other person with legal custody or control of the student.
  8. Participation in a non-school-sponsored extracurricular activity. A school principal or the principal's designee may excuse a student from school attendance to participate in a non-school-sponsored extracurricular activity, if the following conditions are met: (1) The student provides documentation to the school as proof of the student's participation in the non-school-sponsored extracurricular activity; and (2) The student's parent, custodian, or other person with legal custody or control of the student, prior to the extracurricular activity, submits to the principal or the principal's designee a written request for the excused absence. The written request shall be submitted no later than seven (7) business days prior to the student's absence.

The written request shall include:

  1. The student's full name and personal identification number;
  2. The student's grade;
  3. The dates of the student's absence;
  4. The reason for the student's absence; and
  5. The signature of both the student and the student's parent, custodian, or other person with legal custody or control of the student. The principal or the principal's designee shall approve, in writing, the student's participation in the non-school-sponsored extracurricular activity. The principal may limit the number and duration of non-school-sponsored extracurricular activities for which excused absences may be granted to a student during the school year; however, such the principal shall excuse no more than ten (10) absences each school year for students participating in non-school-sponsored extracurricular activities. Students receiving an excused absence under this section shall have the opportunity to make up school work missed and shall not have their class grades adversely affected for lack of class attendance or class participation due to the excused absence.

A written statement within two (2) school days of the student's return to school shall be required from the parent or guardian explaining the reason for each absence. If necessary, verification is required from an official source to justify absences. All absences other than those outlined above shall be considered unexcused.

[School Logo] Trudy XXXXX­XXXXX­XXX
School Counselor
XXXXX­XXXXX Elementary
XXXXX­XXXXX­XXXXX­XXXXX­XX
Memphis, TN 38116
901-XXXXX­XXX (School)
901-XXXXX­XX (Fax)
XXXXX­XXXXX­XXXXX­XXXXX­XX

"Every child deserves to be a champion: an adult who will never give up on them, who understands the power of connection and insists they become the best they can possibly be"-Rita Pierson

Together, we MUST BELIEVE.

Together, we WILL ACHIEVE.

Together, we ARE REIMAGINING 901!

One thing stood out to me—the family name given in the greeting doesn't match my family name, nor does it match any of the family names of any of the other recipients on the email. Otherwise, much like the first email, I have to wonder why I'm receiving this. So I decided to reply to this one with a bit more sarcasm:

From
Sean Conner <sean.conner@gmail.com>
To
Trudy XXXXX­XXXXX­X <XXXXX­XXXXX­XXXXX­XXXXX­XX>
Subject
School Wide Attendance/Chronic Absence IMPORTANT
Date
Wed, 1 Mar 2023 14:55:00 -0500

I must say, it appears that excused absences got more lenient over the years since I was in school. Back when I was in school, excused absences were only allowed with a 10 day prior notice, or appropriate documentation from a doctor, law enforcement officer, or pardon from the governor. What is it with these weak policies towards absences? This is intolerable for my non-existent child in a school three states away!

Here, I expected Trudy to clue in—I mean, “pardon from the governor?” Who ever heard of such a policy for a school? But again, not to be outdone, I got this back from Trudy:

From
Trudy XXXXX­XXXXX­X <XXXXX­XXXXX­XXXXX­XXXXX­XX>
To
Sean Conner <sean.conner@gmail.com>
Subject
RE: EXTERNAL - Re: School Wide Attendance/Chronic Absence IMPORTANT
Date
Web, 1 Mar 2023 14:57:00 -0500

Thank you for your concern, please make sure that you have withdrawn your child from XXXX.

Thank you

Sent from Mail for Windows

…

I … I have no words.

What have I gotten myself into?

Update on Thursday, March 2nd, 2023

A few more emails exchanged, and Trudy and I have straightened things out.

Monday, March 06, 2023

Another attempt at a “unit test”

Or, “What is a ‘unit test,’ part III”

The reactions to my previous post were interesting—it wasn't a “unit test.” At best, it might have been an “integration test” but because it involved actual work (i.e. interaction with the outside world via nasty nasty side effects, aka I/O) it immediately disqualified it as a “unit test.” And to be honest, I was expecting that type of reaction—it appears to me that most unit test proponents tend to avoid such “entanglements” when writing their “battle tested” code (but I'm also willing to admit that's the cynical side of me talking). There were also comments about how 100% code coverage was “unrealistic.”

Sigh.

One respondent even quoted me out of context—“… that we as programmers are not trusted to write code without tests …” and cut the rest of the sentence: “… yet we're trusted to write a ton of code untested as long as such code is testing code.” Which was my cynical take that the “unit tests” (or the code that implements “unit tests” ) are, themselves, not subjected to “unit tests.” Something I kept trying to impart to my former manager, “stop taking the unit tests as gospel! I don't even trust them!” (mainly because the business logic of the project was convoluted and marketing kept using different terms from engineering, at least engineering in my department)

But when I left off, I said there was one final function that should fit as a “unit,” and thus, perfect for “unit testing.” Again, it's from my blog engine and the function in question deals with parsing a request, like “2001/10/02.2-11/03.3” (which requests all blog posts starting from the second post on October 2nd to the third post of November 3rd, 2001). or “2001/11/04.2” (the second post from November 4th, 2001).

The function tumbler_new() does no I/O (that is—no disk, network or console I/O), touches no global variables, only works with the data given to it and does some covoluted parsing of the input data—if this isn't a good candidate for “unit tests” then I don't know what is.

The tests were straightforward—a bunch of failure cases:

  tap_assert(!tumbler_new(&tumbler,"foo/12/04.1",&first,&last),"non-numberic year");
  tap_assert(!tumbler_new(&tumbler,"1999/foo/04.1",&first,&last),"non-numeric month");
  tap_assert(!tumbler_new(&tumbler,"1999/12/foo.1",&first,&last),"non-numeric day");
  tap_assert(!tumbler_new(&tumbler,"1999/12/04.foo",&first,&last),"non-numeric part");
  tap_assert(!tumbler_new(&tumbler,"1998",&first,&last),"before the start year");
  tap_assert(!tumbler_new(&tumbler,"1999/11",&first,&last),"before the start month");
  tap_assert(!tumbler_new(&tumbler,"1999/12/03",&first,&last),"before the start day");
  tap_assert(!tumbler_new(&tumbler,"1999/12/04.0",&first,&last),"part number of 0");
  tap_assert(!tumbler_new(&tumbler,"2023",&first,&last),"after the end year");
  tap_assert(!tumbler_new(&tumbler,"2022/11",&first,&last),"after the end month");
  tap_assert(!tumbler_new(&tumbler,"2022/10/07",&first,&last),"after the end day");
  tap_assert(!tumbler_new(&tumbler,"2022/10/06.21",&first,&last),"after the end part");
  tap_assert(!tumbler_new(&tumbler,"1999/00/04.1",&first,&last),"month of 0");
  tap_assert(!tumbler_new(&tumbler,"1999/13/04.1",&first,&last),"month of 13");
  tap_assert(!tumbler_new(&tumbler,"1999/12/00.1",&first,&last),"day of 0");
  tap_assert(!tumbler_new(&tumbler,"1999/12/32.1",&first,&last),"day of 32");
  tap_assert(!tumbler_new(&tumbler,"1999/12/04.0",&first,&last),"part of 0");
  tap_assert(!tumbler_new(&tumbler,"1999/12/04.24",&first,&last),"part of 24");
  tap_assert(!tumbler_new(&tumbler,"2010/07/01-04/boom.jpg",&first,&last),"file with range");
  tap_assert(!tumbler_new(&tumbler,"2010/7/1-4/boom.jpg",&first,&last),"file with redirectable range");

Plus a bunch of tests that should pass:

  test("first entry","1999/12/04.1",&(tumbler__s) {
    .start    = { .year = 1999 , .month = 12 , .day = 4 , .part = 1 },
    .stop     = { .year = 1999 , .month = 12 , .day = 4 , .part = 1 },
    .ustart   = UNIT_PART,
    .ustop    = UNIT_PART,
    .segments = 0,
    .file     = false,
    .redirect = false,
    .range    = false,
    .filename = ""
  });
  
  test("some mid entry","2010/07/04.15",&(tumbler__s) {
    .start    = { .year = 2010 , .month = 7 , .day = 4 , .part = 15 },
    .stop     = { .year = 2010 , .month = 7 , .day = 4 , .part = 15 },
    .ustart   = UNIT_PART,
    .ustop    = UNIT_PART,
    .segments = 0,
    .file     = false,
    .redirect = false,
    .range    = false,
    .filename = ""
  });
  
  test("last entry","2022/10/06.20",&(tumbler__s) {
    .start    = { .year = 2022 , .month = 10 , .day = 6 , .part = 20 },
    .stop     = { .year = 2022 , .month = 10 , .day = 6 , .part = 20 },
    .ustart   = UNIT_PART,
    .ustop    = UNIT_PART,
    .segments = 0,
    .file     = false,
    .redirect = false,
    .range    = false,
    .filename = ""
  });
  
  test("requesting a file","2010/07/04/boom.jpg",&(tumbler__s) {
    .start    = { .year = 2010 , .month = 7 , .day = 4 , .part =  1 },
    .stop     = { .year = 2010 , .month = 7 , .day = 4 , .part = 23 },
    .ustart   = UNIT_DAY,
    .ustop    = UNIT_DAY,
    .segments = 0,
    .file     = true,
    .redirect = false,
    .range    = false,
    .filename = "boom.jpg",
  });

  /* ... other tests ... */

With this function checking the results:

static void test(char const *tag,char const *tum,tumbler__s const *result)
{
  tumbler__s  tumbler;
  
  assert(tag    != NULL);
  assert(tum    != NULL);
  assert(result != NULL);

  tap_plan(10,"%s: %s",tag,tum);
  tap_assert(tumbler_new(&tumbler,tum,&first,&last),"create");
  tap_assert(btm_cmp(&tumbler.start,&result->start) == 0,"start date");
  tap_assert(btm_cmp(&tumbler.stop,&result->stop) == 0,"stop date");
  tap_assert(tumbler.ustart == result->ustart,"segment of start");
  tap_assert(tumbler.ustop == result->ustop,"segment of stop");
  tap_assert(tumbler.segments == result->segments,"number of segments");
  tap_assert(tumbler.file == result->file,"file flag");
  tap_assert(tumbler.redirect == result->redirect,"redirect flag");
  tap_assert(tumbler.range == result->range,"range flag");
  tap_assert(strcmp(tumbler.filename,result->filename) == 0,"file name");
  tap_done();
}

I ended up with a total of 328 tests and of the three attempts I made, this one feels like the only one that was worth the effort—it's a moderately long function [Moderately long? It's 450 lines long! —Editor] [But it does one thing, and one thing well—it parses a request! —Sean] [450 lines! —Editor] [I'd like to see you write it then! —Sean] [… Okay, I'll shut up now. —Editor] that implements some tricky logic and deal with some weird edge cases. If I ever go back to rework this code (and I've only revised this code once in the 23 years it's been used, way back in 2015—it was a full rewrite of the function) the tests could be useful (if I'm honest with myself, and the API/structure doesn't change). And from looking over the test cases, I can see that I could get rid if .segments from the structure, so there's that.(Seems I was wrong---the .segments field is needed for tumbler_canonical())

Overall, I'm still not entirely sure about this “unit test” stuff, especially since “unit” doesn't have a well defined meaning. In my opinion, I think it works best for functions that do pure processing (no interaction with the outside world) and that implement some complex logic, like parsing or business logic. Back when I was at The Enterprise, had we but one function (or entry point) that just implemented all the business logic from data gathered from the request, it would have have made testing so much easier. But “unit tests” for all functions? Or modules? Or whatever the XXXX a “unit” is? No. Not for obvious code, or for for code that interacts with external systems (unless required because human lives are on the line). I'm not saying no to tests entirely, but to the slavish adherence to testing for its own sake.

Or maybe, instead of having AI write code for us, have it write the test cases for us, intead of the future I'm cynically seeing—where we write the test cases for AI written code.

Discussions about this entry

Tuesday, March 07, 2023

So frustrated that I have no one to scream at, which may be the point why it's so hard to get ahold of customer representatives at tech companies these days

I'm so frustrated right now.

Bunny can't receive email, and we have no idea why that is. All we get is that there have been too many attempts to log into her account and she needs to reset the password. Now, her account is with bellsouth.net which is now owned by AT&T but email for customers is handled by Yahoo, but trying to track down a human being to talk to is a Herculean effort these days, and even if we get ahold of someone, can they even help? Forget the left hand not knowing what the right hand is doing, it appears these days that the left hand doesn't even know it has fingers!

Going to the AT&T login page mentions something about currently.com, but going to currently.com goes to yahoo.com (which, to be fair, is handling email for bellsouth.net). Man, is this confusing. We go through the process of changing the password, only to get redirected back to the login page, which fails because the password we set seems to be incorrect, and thus, we get more and more failed logins until we're forced to change the password yet again.

And the cycle continues.

Doing a search shows that we aren't alone in this, and that this issue has been an ongoing problem for several years now with no solution in sight.

Does anybody at AT&T or Yahoo know what's going on?

To add further fuel, I can't find the current server settings in the Apple Mail application, adding to my frustration.

Eventually, I gave up and set her up with Fastmail. It's not as big as Google, and it's a pay service, so there's a decent chance that talking to a human is possible. Also, it was nearly painless to set up. I say “nearly painless” because it did take me several minutes to figure out the DNS settings I needed to give Bunny her own sub-domain in conman.org (because if she's going through the pain of changing an email address from one she's used for years, I'd rather she use a domain not in the control of the email company in the off chance we need to switch providers so she can keep the same email address, and I don't want to hand off my own email to Fastmail since I'm comfortable using mutt directly on my server). Then there was one file to download that installed information on her Mac laptop that informed the Apple Mail application of the Fastmail settings and that was that. She was good to go.

It just sucks that she has to change her email address.

Monday, March 13, 2023

“We couldn't pause TV and we couldn't just fast forward through the commercials! Gather around kids, and I shall tell more horror tales of the past …”

Over a week ago, I had to order some checks from my bank due to a new recurring expense. My last check I wrote from my checkbook was for some time in 2015. Then one check from 2014, and one from 2013.

Needless to say, I haven't had a need for a checkbook in years.

So the new checkbook arrived (two of them, actually). I recall in the past, they would arrive in the mail in a box sized large enough to hold the checks. But today, they arrived in an unusual format—a flat package about 6½ × 10 × ½ inches (16.5 × 25.5 × 10 mm). Upon opening it, it looked like a book, and stuck to the inside cover were two checkbooks. I pulled them off, and beneath one of them was instructions on how to write a check (with a link to an instructional video)!

I suddenly feel old.

Notes on optimizing an O(n)+C algorithm where the C matters quite a bit

[Note: all hexadecimal values are preceeded with a “$”, which is old-school. I know these days it's hip to use “0x” to designate a hexadecimal value, but I hope this is just a passing fad. Also, the “K” here stands for “kilobyte,” which is 1024 bytes. This too, is old fasioned but I don't want to use a word that sounds like pet food. Geeze, kids these days!] [Here's an onion—put it on your belt, you geezer! —The kids] [Hey! I resemble that remark! Get off my lawn! —Sean]

I was doing a bit of retro computing over the weekend, writing 6809 code and running it on a Color Computer emulator (because the Color Computer was my first computer and the 6809 is a criminially underrated 8-bit CPU in my opinion). Part of the coding was enabling all 64K of RAM in the machine. Normally, the Color Computer only sees the lower 32K of RAM, with the upper 32K being ROM (the BASIC interpreter). To enable all 64K of RAM, all that's needed is to stuff any value into memory location $FFDF, which can be done with “POKE &HFFDF,0”. The problem with that is once the ROM goes away, so does BASIC, and the CPU starts executing Lord knows what since the RAM isn't initialized. So the actual procedure is to copy the ROM contents into RAM, which is simple enough:

	orcc	#$50	; disable interrupts
	ldx	#$8000	; start of ROM
loop	sta	$FFDE	; enable ROM mapping
	lda	,x	; read byte
	sta	$FFDF	; enable RAM
	sta	,x+	; write byte back, increment pointer
	cmpx	#$FF00	; are we done?
	bne	loop
	andcc	#$AF	; enable interrupts

We don't actually want to copy the full 32K of ROM, since the upper 256 bytes of the memory map is for I/O devices. We also disable interrupts since the default interrupt handlers are in ROM and if an interrupt happens when we have RAM mapped, there may not be an interrupt handler to handle it.

The code is straightforward, simple, and unfortunately, slow. Here's the main loop again, this time with the number of cycles and bytes each instruction takes:

			; cycles	bytes
loop1	sta	$FFDE	; 5		3
	lda	,x	; 4		2
	sta	$FFDF	; 5		3
	sta	,x+	; 6		2
	cmpx	#$FF00	; 4		3
	bne	loop1	; 3		2

The loop is 15 bytes, taking 27 cycles for each iteration. Since we copy one byte per iteration and we have 35,512 iterations, it takes 877,824 cycles to run (there are no pipe lines or caches to worry about, so this will always take 877,824 cycles to run). Given the Color Computer runs at .89 MHz (yes, it's not a fast computer) this is nearly a second to copy 32K of RAM.

Can we do better?

Well, yes. Should we is another matter. I mean, this code is typically run once, so does it matter if it takes a second to run? Meh. It'll take longer to load the code from disk, but hey, I'm doing recreational retro programming and want to have a bit of fun.

The first and easiest optimization is to read and write 16 bits at a time instead of 8. And that's easy enough to do—the 6809 does have a 16-bit accumulator so it's an easy change:

			; cycles	bytes
loop2	sta	$FFDE	; 5		3
	ldd	,x	; 5		2
	sta	$FFDF	; 5		3
	std	,x++	; 8		2
	cmpx	#$FF00	; 4		3
	bne	loop2	; 3		2

The loop now takes 30 cycles per iteration, but it's doing 16-bits per iteration instead of 8. The code now takes 487,680 cycles, which is almost half the time, and we've cut the cycles per byte to 15 from 27, and the size of the code hasn't changed. Not bad for a simple optimization.

But doing 4 bytes per iteration should be better, right? It'll take another register (which we have) and some additional bytes, but yes, it is better:

			; cycles	bytes
loop3	sta	$FFDE	; 5		3
	ldd	,x	; 5		2
	ldu	2,x	; 6		2
	sta	$FFDF	; 5		3
	std	,x++	; 8		2
	stu	,x++	; 8		2
	cmpx	#$FF00	; 4		3
	bne	loop3	; 3		2

Each iteration now takes 44 cycles, but we're moving 4 bytes per iteration, giving us 11 cycles per byte, and it takes a total of 357,632 cycles. Again an improvement but we're starting to hit diminishing improvements. Doing 6 bytes per iteration (still easy to add) takes us down to 9.833 cycles per byte, and 8 bytes per iteration takes us down to 9.24 cycles per byte, but we require the use of the S register (the system stack pointer, which needs to be saved and restored) to do so. It's not worth trying to use the last register available to us (DP) because you can't load it directly. The loop also increases in size, from 19 bytes (shown above) to 23 bytes for the 8-bytes-per-iteration version. Also, the upper address will have to change for the 6 byte version, since 6 doesn't cleanly divide 32,512. It's not a show stopper, since not all the memory in the upper 32K contains useful code, so a few bytes not copied won't crash BASIC.

But we can still do better!

Taking inspiration from “A Great Old-Timey Game-Programming Hack” we can copy 8 bytes per iteration (first, the commented code, then the code with the cycles/bytes columns):

	orcc	#$50	; disable interrupts
	sts	savesp	; we need the S register
	lds	#$FF00-8; and because we're using the stack,
			; we need to start at the top of ROM
			; and work our way down
loop4	sta	$FFDE	; enable ROM mapping
	puls	u,x,y,d	; pull 4 2-byte registers from memory (read memory)
	sta	$FFDF	; enable RAM
	pshs	u,x,y,d	; push 4 2-byte registers to memory (write memory)
	leas	-8,s	; point to next 8-byte block to transfer
	cmps	#$8000-8; are we done?
	bne	loop4
	lds	savesp	; restore S register
	andcc	#$AF	; enable interrupts

The PULS instruction pulls the listed registers off the stack, and the PSHS instruction pushes the listed registers onto the stack. The order of registers in the instructions doesn't matter; they get pushed and pulled such that it all works out.

			; cycles	bytes
loop4	sta	$FFDE	; 5		3
	puls	u,x,y,d	; 13		2
	sta	$FFDF	; 5		3
	pshs	u,x,y,d	; 13		2
	leas	-8,s	; 5		2
	cmps	#$8000-8; 5		4
	bne	loop4	; 3		2

The main loop is now 18 bytes, so it's on par with the third version. But each iteration now takes 49 cycles, but given it moves 8 bytes per iteration, we get an effective rate of 6.125 cycles per byte, and a total time of 199,136 cycles. We could do nine bytes per iteration (as the PSHS and PULS instructions support the DP register) to get us down to 5.666 cycles per byte (184,235 cycles total). We could also add in the CC register for a total of 10 bytes per iteration (5.3 cycles per byte; 172,314 cycles total) but we run the risk of enabling interrupts at the wrong time, unless we disable interrupts at the hardware level (which we can do, but it's more code and more invasive of system state). You can see we'd be getting into diminishing returns again, so I'm happy with just doing 8 bytes per iteration.

Summary of results of copying 32,512 bytes from ROM to RAM
loop cycles­/­iteration cycles­/­byte cycles total bytes­/­iteration code size
loop1 27 27.000 877,824 1 15
loop2 30 15.000 487,680 2 15
loop3 44 11.000 357,632 4 19
loop4 49 6.125 199,136 8 18
(theoretical) 53 5.300 172,314 10 18

Is this important these days? Not really. Is it fun? Yes, it certainly beats Enterprise Agile any day of the week.

Discussions about this entry

Wednesday, March 22, 2023

Preloading Lua modules, part III

I received an email from Andy Weidenbaum today, thanking me for writing a post about embedding Lua code into an executable which helped him in his project. On the plus side, it's nice that a post of mine was able to help him. On the non-plus side, I wrote that post ten years ago tomorrow!

Geeze, where does the time go?

I replied to him saying that I have since updated the code (and sent him some copies) but I think I should at least mention it here on the blog. The first major change is populating the Lua array package.preload with the C-based Lua modules (which is probably the intent of that array). The second major change was compressing the Lua-based Lua modules using zlib to save space (and the decompression time on modern systems is near negligible). I accomplish this via a custom tool and the following rule in the makefile:

BIN2C = bin2c

%.o : %.lua
	$(BIN2C) -9 -o $*.l.c -t $(NS)$(*F) $<
	$(CC) $(CFLAGS) -c -o $@ $*.l.c
	$(RM) $*.l.c

This defines an implicit rule to convert a .lua file into a .o (object) file that can be linked. The first line will read the Lua source file, compress it (via the -9 option for heaviest compression) and convert it to a C file. The C file is then compiled into an object file, and the C file is then removed as it's no longer needed.

The third major change is the routine now expects to initialize the entire Lua state, set up the embedded C and Lua based modules, then run the Lua “application” by name (I went from making a Kitchen Sink Lua interpreter to stand alone apps mainly written in Lua).

There are other minor changes, but I think at this point it's best to show some code:

#include <stdlib.h>
#include <string.h>
#include <assert.h>

#include <zlib.h>

#include <lua.h>
#include <lauxlib.h>
#include <lualib.h>

#if LUA_VERSION_NUM == 501
#  define lua_rawlen(L,idx)       lua_objlen((L),(idx))
#  define luaL_setfuncs(L,reg,up) luaI_openlib((L),NULL,(reg),(up))
#  define SEARCHERS               "loaders"
#  define lua_load(L,f,d,k,x)     (lua_load)((L),(f),(d),(k))
#else
#  define SEARCHERS               "searchers"
#endif

/**************************************************************************/

typedef struct prelua_reg
{
  char          const *const name;
  unsigned char const *const code;
  size_t        const *const size;
} prelua_reg__s;

struct zlib_data
{
  char const *name;
  z_stream    sin;
  Bytef       buffer[LUAL_BUFFERSIZE];
};

/**************************************************************************/

static char const *preloadlua_reader(lua_State *L,void *ud,size_t *size)
{
  assert(L    != NULL);
  assert(ud   != NULL);
  assert(size != NULL);
  
  struct zlib_data *data = ud;
  (void)L;
  
  data->sin.next_out  = data->buffer;
  data->sin.avail_out = sizeof(data->buffer);
  
  inflate(&data->sin,Z_SYNC_FLUSH);
  
  *size = sizeof(data->buffer) - data->sin.avail_out;
  return (char const *)data->buffer;
}

/*************************************************************************/

static int preload_lualoader(lua_State *const L)
{
  char          const *key = luaL_checkstring(L,1);
  prelua_reg__s const *target;
  
  assert(L != NULL);
  
  lua_getfield(L,LUA_REGISTRYINDEX,"org.conman:prelua");
  lua_getfield(L,-1,key);
  target = lua_touserdata(L,-1);
  
  if (target == NULL)
  {
    lua_pushfstring(L,"\n\tno precompiled module '%s'",key);
    return 1;
  }
  else
  {
    struct zlib_data data;
    
    data.sin.zalloc   = Z_NULL;
    data.sin.zfree    = Z_NULL;
    data.sin.opaque   = Z_NULL;
    data.sin.next_in  = (Byte *)target->code;
    data.sin.avail_in = *target->size;
    
    inflateInit(&data.sin);
    lua_load(L,preloadlua_reader,&data,key,NULL);
    inflateEnd(&data.sin);
    lua_pushliteral(L,":preload:");
    return 2;
  }
}

/***********************************************************************/

int exec_lua_app(
        lua_State           *L,
        char          const *modname,
        luaL_Reg      const *preload,
        prelua_reg__s const *prelua,
        int                  argc,
        char               **argv
)
{
  int preluasize;
  
  assert(L       != NULL);
  assert(modname != NULL);
  assert(preload != NULL);
  assert(prelua  != NULL);
  assert(argc    >  0);
  assert(argv    != NULL);
  
  for (preluasize = 0 ; prelua[preluasize].name != NULL ; preluasize++)
    ;
    
  lua_createtable(L,0,preluasize);
  
  for (int i = 0 ; i < preluasize ; i++)
  {
    lua_pushlightuserdata(L,(void *)&prelua[i]);
    lua_setfield(L,-2,prelua[i].name);
  }
  
  lua_setfield(L,LUA_REGISTRYINDEX,"org.conman:prelua");
  luaL_openlibs(L);
  lua_getglobal(L,"package");
  lua_getfield(L,-1,"preload");
  luaL_setfuncs(L,preload,0);
  lua_getglobal(L,"package");
  lua_getfield(L,-1,SEARCHERS);
  
  for (lua_Integer i = lua_rawlen(L,-1) + 1 ; i > 1 ; i--)
  {
    lua_rawgeti(L,-1,i - 1);
    lua_rawseti(L,-2,i);
  }
  
  lua_pushinteger(L,2);
  lua_pushcfunction(L,preload_lualoader);
  lua_settable(L,-3);
  lua_pop(L,4);
  
  lua_createtable(L,argc,0);
  for (int i = 0 ; i < argc ; i++)
  {
    lua_pushinteger(L,i);
    lua_pushstring(L,argv[i]);
    lua_settable(L,-3);
  }
  lua_setglobal(L,"arg");
  
  lua_pushcfunction(L,preload_lualoader);
  lua_pushstring(L,modname);
  lua_call(L,1,1);

  return lua_pcall(L,0,0,0);
}

And to show how it's used, I'll use my gopher server as an example:

int main(int argc,char *argv[])
{
  static luaL_Reg const c_preload[] =
  {
    { "lpeg"                  , luaopen_lpeg                  } ,
    { "org.conman.clock"      , luaopen_org_conman_clock      } ,
    { "org.conman.errno"      , luaopen_org_conman_errno      } ,
    { "org.conman.fsys"       , luaopen_org_conman_fsys       } ,
    { "org.conman.fsys.magic" , luaopen_org_conman_fsys_magic } ,
    { "org.conman.math"       , luaopen_org_conman_math       } ,
    { "org.conman.net"        , luaopen_org_conman_net        } ,
    { "org.conman.pollset"    , luaopen_org_conman_pollset    } ,
    { "org.conman.process"    , luaopen_org_conman_process    } ,
    { "org.conman.signal"     , luaopen_org_conman_signal     } ,
    { "org.conman.syslog"     , luaopen_org_conman_syslog     } ,
    { "port70.getuserdir"     , luaopen_port70_getuserdir     } ,
    { "port70.setugid"        , luaopen_port70_setugid        } ,
    { NULL                    , NULL                          }
  };

  static prelua_reg__s const c_prelua[] =
  {
    { "org.conman.const.exit"            , c_org_conman_const_exit            , &c_org_conman_const_exit_size            } ,
    { "org.conman.const.gopher-types"    , c_org_conman_const_gopher_types    , &c_org_conman_const_gopher_types_size    } ,
    { "org.conman.net.ios"               , c_org_conman_net_ios               , &c_org_conman_net_ios_size               } ,
    { "org.conman.nfl"                   , c_org_conman_nfl                   , &c_org_conman_nfl_size                   } ,
    { "org.conman.nfl.tcp"               , c_org_conman_nfl_tcp               , &c_org_conman_nfl_tcp_size               } ,
    { "org.conman.parsers.abnf"          , c_org_conman_parsers_abnf          , &c_org_conman_parsers_abnf_size          } ,
    { "org.conman.parsers.ascii.char"    , c_org_conman_parsers_ascii_char    , &c_org_conman_parsers_ascii_char_size    } ,
    { "org.conman.parsers.ascii.control" , c_org_conman_parsers_ascii_control , &c_org_conman_parsers_ascii_control_size } ,
    { "org.conman.parsers.ip-text"       , c_org_conman_parsers_ip_text       , &c_org_conman_parsers_ip_text_size       } ,
    { "org.conman.parsers.iso.char"      , c_org_conman_parsers_iso_char      , &c_org_conman_parsers_iso_char_size      } ,
    { "org.conman.parsers.iso.control"   , c_org_conman_parsers_iso_control   , &c_org_conman_parsers_iso_control_size   } ,
    { "org.conman.parsers.mimetype"      , c_org_conman_parsers_mimetype      , &c_org_conman_parsers_mimetype_size      } ,
    { "org.conman.parsers.url"           , c_org_conman_parsers_url           , &c_org_conman_parsers_url_size           } ,
    { "org.conman.parsers.url.gopher"    , c_org_conman_parsers_url_gopher    , &c_org_conman_parsers_url_gopher_size    } ,
    { "org.conman.parsers.utf8.char"     , c_org_conman_parsers_utf8_char     , &c_org_conman_parsers_utf8_char_size     } ,
    { "org.conman.parsers.utf8.control"  , c_org_conman_parsers_utf8_control  , &c_org_conman_parsers_utf8_control_size  } ,
    { "port70"                           , c_port70                           , &c_port70_size                           } ,
    { "port70.cgi"                       , c_port70_cgi                       , &c_port70_cgi_size                       } ,
    { "port70.handlers.content"          , c_port70_handlers_content          , &c_port70_handlers_content_size          } ,
    { "port70.handlers.file"             , c_port70_handlers_file             , &c_port70_handlers_file_size             } ,
    { "port70.handlers.filesystem"       , c_port70_handlers_filesystem       , &c_port70_handlers_filesystem_size       } ,
    { "port70.handlers.http"             , c_port70_handlers_http             , &c_port70_handlers_http_size             } ,
    { "port70.handlers.sample"           , c_port70_handlers_sample           , &c_port70_handlers_sample_size           } ,
    { "port70.handlers.url"              , c_port70_handlers_url              , &c_port70_handlers_url_size              } ,
    { "port70.handlers.userdir"          , c_port70_handlers_userdir          , &c_port70_handlers_userdir_size          } ,
    { "port70.mklink"                    , c_port70_mklink                    , &c_port70_mklink_size                    } ,
    { "port70.readfile"                  , c_port70_readfile                  , &c_port70_readfile_size                  } ,
    { "port70.safetext"                  , c_port70_safetext                  , &c_port70_safetext_size                  } ,
    { "re"                               , c_re                               , &c_re_size                               } ,
    { NULL                               , NULL                               , NULL                                     }
  };
  
  lua_State *L = luaL_newstate();
  if (L == NULL)
  {
    fprintf(stderr,"Cannot create Lua state\n");
    return EXIT_FAILURE;
  }
  
  int rc = exec_lua_app(L,"port70",c_preload,c_prelua,argc,argv);
  if (rc != LUA_OK)
    fprintf(stderr,"lua_pcall() = %s\n",lua_tostring(L,-1));
    
  lua_close(L);
  return rc == LUA_OK ? EXIT_SUCCESS : EXIT_FAILURE;
}

I haven't yet written code to automagically write the main() routine, but for a small application like port70 it wasn't that bad. There are several approaches to automating this, like overriding require() with some custom code to record loaded modules, or building the list from LuaRocks, but that's a project for another day.

Obligatory Picture

[The future's so bright, I gotta wear shades]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: https://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

https://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.