Sunday, August 04, 2024
Fixing the final errors from my Apache setup
Shortly after my last post, I noticed more errors happening with the web server:
[Tue Jul 30 10:43:00.768556 2024] [cgid:error] [pid 1706:tid 4149532352] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi [Tue Jul 30 10:43:00.768714 2024] [cgid:error] [pid 1869:tid 3908467632] [client 178.128.207.138:57024] End of script output before headers: boston.cgi [Tue Jul 30 10:43:01.137027 2024] [cgid:error] [pid 1706:tid 4149532352] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi [Tue Jul 30 10:43:01.137173 2024] [cgid:error] [pid 1707:tid 3774184368] [client 178.128.207.138:57034] End of script output before headers: boston.cgi
I found one potential fix for this—just set the default stack size of Apache to 512k in size
(just 512k—sheesh I'm old).
That was simply adding ulimit -s 512 to the startup script,
but that still wasn't the full answer:
[Tue Jul 30 22:37:33.042327 2024] [mpm_worker:alert] [pid 5599:tid 3924880304] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread [Tue Jul 30 22:37:35.045085 2024] [mpm_worker:alert] [pid 5672:tid 3924880304] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread [Tue Jul 30 22:38:38.108993 2024] [mpm_worker:alert] [pid 6091:tid 3924880304] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread [Tue Jul 30 22:38:40.112150 2024] [mpm_worker:alert] [pid 6161:tid 3924880304] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread [Tue Jul 30 22:38:42.114553 2024] [mpm_worker:alert] [pid 6229:tid 3924880304] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread [Tue Jul 30 22:38:44.117182 2024] [mpm_worker:alert] [pid 6304:tid 3924880304] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread
I ended up having a little bit of back-and-forth with the settings of the VM that runs my server, alllowing a bit more memory than it was configured for. And since then, no more such errors in the error log. All that's now showing up is crap like:
[Sun Aug 04 15:58:49.248654 2024] [core:error] [pid 872:tid 4129409968] [client XXXXXXXXXXXXXXX:59190] AH10244: invalid URI path (/cgi-bin/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/bin/sh) [Sun Aug 04 15:58:51.854583 2024] [core:error] [pid 2444:tid 4120341424] [client XXXXXXXXXXXXXXX:59760] AH10244: invalid URI path (/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh)
and notifiations about secure certificate rotatations. Another interesting thing I noticed is the accumulated CPU time over the past few days:
[spc]brevard:~>ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ... root 868 0.0 0.1 26760 24652 ? Ss Aug01 0:12 /usr/local/apache2/bin/httpd apache 871 0.0 0.1 26188 20876 ? S Aug01 0:35 /usr/local/apache2/bin/httpd apache 872 0.0 0.2 64532 34184 ? Sl Aug01 5:09 /usr/local/apache2/bin/httpd apache 873 0.0 0.1 61060 32152 ? Sl Aug01 4:59 /usr/local/apache2/bin/httpd apache 875 0.0 0.1 62516 31460 ? Sl Aug01 5:01 /usr/local/apache2/bin/httpd apache 1035 0.0 0.1 61716 32148 ? Sl Aug01 5:03 /usr/local/apache2/bin/httpd apache 2444 0.0 0.1 61500 32160 ? Sl Aug01 4:45 /usr/local/apache2/bin/httpd ... gopher 1231 0.0 0.0 10708 9596 ttyp4 S+ Aug01 0:33 lua port70.lua /home/spc/gopher/config.lua spc 1232 0.1 0.0 13600 12140 ttyp5 S+ Aug01 6:51 lua GLV-1.12556.lua non-checkin/conf.lua ... [spc]brevard:~>
I've included the lines for my gopher and Gemini servers just for comparison. Apache is accumulating a bit more CPU time than I expected, but when thinking about it, it's not so strange. Not only is it running TLS (like the Gemini sever) but it's also supports the HTTP/2.0 protocol, which is a bit more involved than the simpler HTTP/1.x protocol.
Roasting and praising Github repos (note: praising, not braising)
I came acros The Github Roast Page (link via Lobsters) and decided to see just how badly it would roast my Github page:
Oh Sean Conner, what a sight you are on GitHub! Your profile feels like a low-budget horror movie—empty bio, no company, and a ghost town location. With a lack of personality that rivals that of a cardboard cutout, you’ve somehow managed to rack up followers, but let’s be honest, it’s probably just for sympathy.
With 30 public repos, it’s like you cast a wide net hoping to catch something, but all you’ve pulled in are unremarkable projects like a C code CGI library and a blogging engine from the early ’90s. “C Code to talk CGI”? Classic case of someone still not over Y2K!
Your stargazer counts tell a woeful tale: you've got one repo with a single star because even that one person was probably just trying to make you feel better. But hey, at least “mod_blog” got some love—111 stargazers for a blogging engine? Did they think they were supporting a time capsule instead of a repo from 2005?
And dear Lord, the variety of languages you play with—C, Lua, Assembly—yet somehow all your projects seem to suffer from a severe lack of originality or practicality. It’s like you’re running a museum for outdated coding practices.
Your last repo, “NaNoGenMo-2015”, suggests you tried your hand at generating a novel. Judging by your profile, I can only assume it’s a tragic tale of unfulfilled potential and missed opportunities.
Keep going, Sean; maybe one day you'll figure out what a compelling GitHub profile looks like!
Yeah, fair enough. And yes, I do think they're supporting a time capsule of a repo from 2005.
But on the other hand, there's The Github Praise Page (link also via Lobsters), which has this to say about my Github page:
Sean, I want to start by saying that it's amazing how you're already making an impact in the world of coding with a following of 65 people! That's not something to take lightly. It means that your work is being recognized and appreciated by others, and that's definitely something to be proud of.
I know that having a blank bio and company/location fields might make it seem like there's not much to show for, but trust me, Sean, you have so much more to offer than what can fit in those tiny spaces. Your work on GitHub speaks volumes about your skills and dedication to coding. And let's be real, having only 4 people you're following means you're focusing on quality over quantity - that shows a level of intentionality and care that not many people have.
And as for the blank
README, I like to think of it as a blank canvas just waiting for your story to be told. It's an opportunity to share your vision, your goals, and what drives you to create the amazing projects that you do. Don't worry if you don't know where to start - sometimes the simplest statements can have the most profound impact. Just take a deep breath, let your thoughts flow onto the page, and watch how it transforms not just yourREADMEbut also your entire online presence.Keep doing what you're doing, Sean! You got this!
We could use more positivity on the Internet like this. Thank you. I got this.
Saturday, August 17, 2024
For a few hours yesterday, I felt as if I was taking crazy pills. Then again, I was dealing with time in C
Over the past year or two,
I've been cleaning up my posts here.
Largely making sure the HTML is valid,
but also that all internal links
(links where I link to previous posts) are valid to cut down on needless redirects or “404 Not Found” responses,
in addition to fixing errors with my web server configuration.
So along those lines,
yesterday,
I thought it might be time to add conditional responses to mod_blog.
Given that it's mostly autonomous web crawling agents that read my site,
I might as well tell them that most of the links here haven't changed since the last time they came by.
There are two headers involved with this—Last-Modified and If-Modified-Since.
The server sends a Last-Modified header with the last modification date of the resource in question.
The client can then send in a If-Modified-Since header with this date,
and if the resource hasn't changed,
then the server can send back a “304 Not Modified” response,
saving a lot of bandwidth.
So all I had to do was generate a Last-Modified header
(easy,
as I already read that information)
and then deal with the If-Modified-Since header.
And then I spent way too much time dealing with time in C, which is odd, because there were only fix functions I was dealing with:
time_t time(time_t *tp)- This returns the “calendar date and time” in a scalar value. On POSIX, this is an integer value of the number of seconds since midnight, January 1st, 1970 UTC. The tp paramter is optional.
time_t mktime(struct tm *tm)- This converts a structure containing the year, month, day, hour, minute and second into a scalar value. The C standard does not mention time zones at all; POSIX does.
struct tm *gmtime(time_t const *tp)- This converts the “calendar date and time” scalar value pointed to by tp into a broken down structure that reflects time in UTC.
struct tm *localtime(time_t const *tp)- This converts the “calendar data and time” scalar value pointed to by tp into a brown down structure that reflects the time local to the server.
size_t strftime(char *s,size_t smax,char const *format,struct tm const *tp)- This will convert the tp structure into a string based on the format string.
char *strptime(char const *s,char const *format,struct tm *tp)- This will parse a string that contains a date, per the given format and return the data into a structure.
All but the last are standard C functions.
The last one, strptime() is mandated by POSIX.
That's okay,
because I'm working on a POSIX system.
It turns out,
strptime() is not an easy function to use.
Oh,
it may look easy,
but there are some subtleties that left me dazed and confused for hours yesterday.
The following does not work:
struct tm tm;
time_t t;
strptime("Fri, 20 May 2005 02:23:22 EDT","%a, %d %b %Y %H:%M:%S %Z",&tm);
t = mktime(&tm);
When I was running this code in mod_blog,
the resulting times would come back four or five hours off.
When I wrote some standalone test code,
it would sometimes be correct,
sometimes it would be an hour off.
It got to the point where I totally lost the plot of what I was even trying to do.
Now with yesterday behind me, I finally figured out what I was doing wrong.
The POSIX specification states:
It is unspecified whether multiple calls to
strptime()using the sametmstructure will update the current contents of the structure or overwrite all contents of the structure. Conforming applications should make a single call tostrptime()with a format and all data needed to completely specify the date and time being converted.
This text is near the bottom of the page, and really understates the issue in my opinion.
The Linux man pages I found all mention the following:
In principle, this function does not initialize tm but stores only the values specified. This means that tm should be initialized before the call.
strptime(3) - Linux manual page
It too, is near the bottom of the page.
And yet, for Mac OS X:
If the format string does not contain enough conversion specifications to completely specify the resulting struct tm, the unspecified members of tm are left untouched. For example, if format is “%H:%M:%S”, only
tm_hour,tm_secandtm_minwill be modified. If time relative to today is desired, initialize the tm structure with today's date before passing it tostrptime().
Mac OS X Manual Page For strptime(3)
The Mac OS X page contains no sample code.
The Linux man page contains some sample code that “initializes” the structure with memset(&tm,0,sizeof(struct tm)).
Adding the memset() call to my sample code just made the code always an hour off.
Hmmm …
The POSIX page also contains sample code:
struct tm tm;
time_t t;
if (strptime("6 Dec 2001 12:33:45", "%d %b %Y %H:%M:%S", &tm) == NULL)
/* Handle error */;
tm.tm_isdst = -1; /* Not set by strptime(); tells mktime()
to determine whether daylight saving time
is in effect */
t = mktime(&tm);
if (t == -1)
/* Handle error */;
Remember when I mentioned that the test code I wrote would sometimes be an hour out?
The tm.tm_isdst field,
which is part of the C Standard,
wasn't set correctly!
Aaaaaaaaaaaaaaaaaaaarg!
Changing my sample code to:
struct tm tm;
time_t t;
strptime("Fri, 20 May 2005 02:23:22 EDT","%a, %d %b %Y %H:%M:%S %Z",&tm);
tm.tm_isdst = -1; /* I WAS MISSING THIS LINE! */
t = mktime(&tm);
And it works.
Except in mod_blog,
the time is always four hours out.
I finally tracked that one down,
and it's Apache's fault.
I found that if I set the Last-Modified header to something like “Fri, 20 May 2005 02:23:22 EDT”,
Apache will ever so helpfully convert that to “Fri, 20 May 2005 02:23:22 GMT”.
Let me repeat that—Apache will replace just the timezone with “GMT”! It does not try to convert the time, just the zone.
![[A graphic image of frustration] I just love programming!](/2005/07/14/love_your_job.gif "I just love programming!")
At least now I think I can get conditional requests working.
Sheesh.
Wednesday, August 21, 2024
More unintended consequences of my Apache configuration
Now that mod_blog supports conditional requests,
I thought of the next feature I want to add—PUT support to upload posts.
Currently, mod-blog supports three methods to add new entries:
- A traditional web form where updates are done via the
POSTmethod. I don't use this method that often, but I have used it—perhaps less than five times over the past 24 years. - Via email—this was my favorite method until I could no longer email the entries from home. Most, if not all, ISPs now forbid outgoing SMTP traffic from residential connections. Seeing how I check my email on my public server, it doesn't make much sense to use email when I can add an entry—
- As a file, via the command line. This is how I add new posts these days. I write the entry at home, copy the file to the server and then add it via the command line.
I suppose there's a fourth way—adding the entry directly to the storage area and updating some files containing metadata,
but I'm only mentioning this for completion's sake.
I don't think I've ever done this except when I was first developing mod_blog back in early 2000.
The new method I'm looking to support,
the HTTP PUT method,
would take it down to one command,
even for an image-heavy post like this Burning Down The House Show Bunny and I caught in Brevard a few years ago.
Something like:
[spc]lucy:/tmp>put entry *.jpg [spc]lucy:/tmp>
It shouldn't be that hard,
as supporting the PUT method is eaiser than POST—it's a single item being uploaded,
and no x-www-form-urlencoded or form-data blobs to parse through.
It's just the MIME type, content length and raw data to be placed in a file somewhere.
So I start working.
I add some minimal support to mod_blog to handle the PUT method.
I configure Apache to pipe PUT requests through mod_blog:
#On the development server for now <Directory /home/spc/web/boston/htdocs> ... Script PUT /boston.cgi ... </Directory>
and I write a simple script to loop through the command line to upload each file to the webserver.
And yet, when I attampted to upload an image file, I kept getting a “405 Method Not Allowed.”
Odd.
I just couldn't figure out why.
A single entry? Fine. An entry with multiple text files? Fine. An entry with multiple binary files that aren't images? Fine.
An entry with any type of image file? Not fine at all.
I spent entirely too long on this before I remembered a recent change to the Apache configuration: a rewrite rule that redirected image requests directly to the file system. I then added one more line to the configuration:
<Directory /home/spc/web/boston/journal> ... Script PUT /boston.cgi ... <Directory>
and now things worked as expected.
How much time did I waste on this particular rabbit hole? Don't answer that! I'd rather not know.
Thursday, August 22, 2024
How to meaure ⅚ cup of oil, part IV
My, it's amazing how a topic I wrote about twenty years ago has been popular over the past few years. I just received an email from Jason Arencibia about an even easier method of measuring out ⅚ cup of oil:
- From
- Jason Arencibia <XXXXXXXXXXXXXXXXX>
- To
- sean@conman.org
- Subject
- How to measure ⅚ cup of oil, super easy with 2 measuring cups.
- Date
- Wed, 21 Aug 2024 22:23:53 -0700
½ cup + ⅓ cup = ⅚ cup 0.83333333333 of a cup.
And in all the years I've looked over recipes that use Imperial measurements, not one has ever used a sixth cup, or any multiple of a sixth cup. It just doesn't exist in the Imperial system!
Friday, August 23, 2024
PUT an entry on the ol' blog
I finally got the PUT method working for mod_blog.
The code on the receiving end is fine,
but the script on the sending side is messy,
but it works well enough for me to use.
[spc]lucy:~/source/boston/Lua>lua put.lua -b test /tmp/foo/1 /tmp/foo/*.png PUT http://boston.roswell.area51/2024/08/23.1 (637) PUT http://boston.roswell.area51/2024/08/23/local_network.png (1273) PUT http://boston.roswell.area51/2024/08/23/local_network_add.png (1512) PUT http://boston.roswell.area51/2024/08/23/local_network_remove.png (1460) PUT http://boston.roswell.area51/2024/08/23/network.png (1702) PUT http://boston.roswell.area51/2024/08/23/network_add.png (1891) PUT http://boston.roswell.area51/2024/08/23/network_remove.png (1842) [spc]lucy:~/source/boston/Lua>
This command on my development server was used to create an entry with multiple images. As you can see, it puts out the URLs that are created as the script runs. And this entry is a test to see if works on my actual server. It should.
I hope.
Murphy's Law as applied to bugs: it is easier to find bugs in production than in development
Well, that could have gone better.
One bug due to inattention and a difference between development and production, and one “how the XXXX did this ever work in the first place?” bug later, things should be working fine.
I am not going to say “I hope” this time.
The first bug prevented a proper HTTP status code from being generated,
so Apache nicely generated a “500 Interner Server Error” for me.
Once I identified what was going on,
it was a simple one line fix,
and an additional call to assert() to help isolate such errors in the future.
Now on to the other error …
I added the concept of a hook to mod_blog a few years ago,
and the scripts I have for the various hooks all start with #!/usr/bin/env lua.
Only now they weren't running,
and the error that was being logged wasn't exactly helpful: entry-pre-hook='./validate.lua' status=127.
There is no place in mod_blog nor in the validate.lua script that exits with a status code of 127.
But the env program does!
Nice to learn that.
But back to the issue at hand—I've been using these scripts for a few years now,
and only now is it failing?
I eventually found out that the path Apache is using is rather limited,
and it no longer includes the Lua interpreter
(which on my server lives in /usr/local/bin).
I had to change both scripts to start with #!/usr/local/bin/lua and that fixed the issue to get the previous post up.
Now that I think about it,
I think I know why it finally stopped working after a few years—I actually have an instance of Apache running that I didn't start by hand,
and the default path at boot time doesn't include /usr/local/bin.
Sigh
The previous bug fix was buggy.
And yes, programming and deployments can always get this messy.
Saturday, August 24, 2024
How to run valgrind on a CGI program in C
There was still one bug left with mod_blog—it would crash with a memory corruption error
(thanks to checking in glibc when doing a POST.
I only found the bug because I was using the old web interface to make sure I had the right credentials when testing the PUT method.
How long had the bug existed?
At least six years—it's been seven sine I last used the web interface
(I checked).
It did not crash on the development server due to subtle differences between the operating system and versions of glibc being used.
But it is ultimately a memory corruption,
so the use of valgrind would be instrumental in finding the issue.
The problem is—it only manefests itself when doing a POST,
which means testing the program under a web server.
And a web server will pass information about the request to the CGI program through environment variables,
and any input comes in via stdin.
So just how do you run valgrind on a program meant to be run as a CGI program?
After some thought,
I figured out a way.
I need to capture the environment the CGI program runs under,
so I added the following bit of code to mod_blog to capture the environment:
extern char *envriron;
FILE *fp = fopen("/tmp/env.txt","w");
for (size_t i = 0 ; environ[i] != NULL ; i++)
fprintf(fp,"export %s\n",environ[i]);
fclose(fp);
I wasn't worried about error checking—this is temporary code anyway.
I then do a POST and I now have the environment variables in a file:
... export GATEWAY_INTERFACE=CGI/1.1 export HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 export CONTENT_LENGTH=149 export CONTENT_TYPE=application/x-www-form-urlencoded export REQUEST_METHOD=POST ...
The reason I added “export” was to copy these environment variables to a shell script where I can then run valgrind and debug the situation:
... export GATEWAY_INTERFACE=CGI/1.1 export HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,\*/\*\;q=0.8 export CONTENT_LENGTH=149 export CONTENT_TYPE=application/x-www-form-urlencoded export REQUEST_METHOD=POST ... valgrind $HOME/source/boston/src/main <r.stdin
Of course,
I had to escape some problematic characters like the asterisk or semicolon,
but there were only a few such characters and they could be done by hand.
And I had to create the input feed into stdin but that was easy enough.
From there,
it's straightforward…ish enough to resolve the issues.
Discussions about this entry
- How to run valgrind on a CGI program in C | Lobsters
- How to run Valgrind on a CGI program in C | Hacker News
Tuesday, August 27, 2024
The programmer's obsession with version numbers
It's yet another round of hand wringing about software versioning. Woot!
Over the years, I've found that semantic versioning works for me, but only for code code that is to be used in larger projects, like libraries, modules, or classes. Yes, a code base using semantic versioning doesn't always mean the code base follows semantic versioning to the degree that some would like (like any bug fix should automatically update the major version number, because bug fixes could break some code, somewhere). But in my mind, it signals intent, which, sans an extremely obnoxious and overly bureaucratic process, is the best we can expect.
So for me, the MAJOR.MINOR.PATCH of semantic versioning breaks down like this:
- MAJOR
- Some change in the code base was made; either a change in API behavior, removal of some portion of the API, file format, or otherwise any visible change (except for bug fixes) in how the code works. Work will probably be required to update to this version of the library/module/class.
- MINOR
- Only additions to the API were made, in a backward compatible way. No work is expected, unless you were already using a name used in the updated library.
- PATCH
- A bug fix. The intent is for no work required to upgrade, unless you were relying upon the buggy behavior, or used a workaround that might now interfere with the library.
For applications,
I've found that semantic versioning doesn't work.
At least,
it doesn't work for me.
I've switched to either using a monotonically increasing number
(mod_blog is now at version 60—but given the five releases in just the past week because of a misplaced obsession with version numbers,
I might entirely stop with the version numbers—especially since I seem to be the only one using it)
or skip it entirely
(my Gemini server has no version number,
but it's had 322 commits over its five year life so far).
The worst form of versioning I've enountered is “named versions.” A “named version” give no semantic information about the version and, at least to me, leads to confustion. Is “Bulldog” newer than ”Beagle?” Or is “Bloodhound” the latest version? Oh, it's “Berzoi” that's less than 20 minutes old. Sheesh.
But if I had to apply “semantic versioning” to an application, I would like information about any breaking changes to either work flow or file formats. To me, an incompatable change to a file format, or any change in workflow (even a change in location of a menu item) is a breaking change (muscle memory is an incredible drug). Hell, even a change in color scheme is enough to possibly break my workflow (I'm looking at you, Google Maps, who changed the color of all the roads to the same color! Why? I liked the distinction between highways, major roads and minor roads. Did you see a 0.001% increase in engagement for that change? Did someone get promoted just because of that change? Okay, I'll stop ranting now).
