The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Thursday, July 04, 2024

Something something Independence something something Declaration something something Blow Stuff Up

It just now became the Forth of July™ and earlier this evening (which was technically yesterday, Wednesday, July 3rd) some people were illegally lighting off fireworks already. I guess as a random quality test of the munitions they've aquired. So to them, and to you, I wish you a happy and safe National Blow Stuff Up Day!

[Amateurs: There's a reason professionals exist.]

And for the curious, the above image was taken by me 21 years ago. Yikes!

And it was 13 years ago when my hair nearly caught on fire, so when I say “be safe,” I mean it.

Sunday, July 21, 2024

Fixing an Apache pthread error

After I fixed the performance issue, I started looking around for some other issues to handle, and boy, did I find some. Checking the error log from Apache I found:

[Fri Jul 12 15:04:01.762845 2024] [mpm_worker:alert] [pid 31979:tid 3924761520] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread
libgcc_s.so.1 must be installed for pthread_cancel to work
[Fri Jul 12 15:04:02.731958 2024] [core:notice] [pid 19646:tid 4149040832] AH00052: child pid 31979 exit signal Aborted (6)
[Fri Jul 12 15:04:02.735360 2024] [mpm_worker:alert] [pid 32021:tid 3924761520] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread
libgcc_s.so.1 must be installed for pthread_cancel to work
[Fri Jul 12 15:04:03.733536 2024] [core:notice] [pid 19646:tid 4149040832] AH00052: child pid 32021 exit signal Aborted (6)
[Fri Jul 12 15:04:03.736857 2024] [mpm_worker:alert] [pid 32063:tid 3924761520] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread
libgcc_s.so.1 must be installed for pthread_cancel to work
[Fri Jul 12 15:04:04.735368 2024] [core:notice] [pid 19646:tid 4149040832] AH00052: child pid 32063 exit signal Aborted (6)
[Fri Jul 12 15:04:04.738624 2024] [mpm_worker:alert] [pid 32105:tid 3924761520] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread
libgcc_s.so.1 must be installed for pthread_cancel to work
[Fri Jul 12 15:04:05.737141 2024] [core:notice] [pid 19646:tid 4149040832] AH00052: child pid 32105 exit signal Aborted (6)
[Fri Jul 12 15:04:05.740622 2024] [mpm_worker:alert] [pid 32147:tid 3924761520] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread
libgcc_s.so.1 must be installed for pthread_cancel to work
[Fri Jul 12 15:04:06.739077 2024] [core:notice] [pid 19646:tid 4149040832] AH00052: child pid 32147 exit signal Aborted (6)
[Fri Jul 12 15:04:06.742500 2024] [mpm_worker:alert] [pid 32189:tid 3924761520] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread
libgcc_s.so.1 must be installed for pthread_cancel to work
[Fri Jul 12 15:04:07.740898 2024] [core:notice] [pid 19646:tid 4149040832] AH00052: child pid 32189 exit signal Aborted (6)
[Fri Jul 12 15:04:07.744130 2024] [mpm_worker:alert] [pid 32231:tid 3924761520] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread
libgcc_s.so.1 must be installed for pthread_cancel to work
[Fri Jul 12 15:04:08.742689 2024] [core:notice] [pid 19646:tid 4149040832] AH00052: child pid 32231 exit signal Aborted (6)
[Fri Jul 12 15:04:08.745885 2024] [mpm_worker:alert] [pid 32274:tid 3924761520] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread
libgcc_s.so.1 must be installed for pthread_cancel to work
[Fri Jul 12 15:04:09.744445 2024] [core:notice] [pid 19646:tid 4149040832] AH00052: child pid 32274 exit signal Aborted (6)
[Fri Jul 12 15:04:09.747607 2024] [mpm_worker:alert] [pid 32316:tid 3924761520] (12)Cannot allocate memory: AH03142: ap_thread_create: unable to create worker thread
libgcc_s.so.1 must be installed for pthread_cancel to work

Thousands of such lines. And the weird thing is that libgcc_s.so.1 does exist on my system. I guess I missed that library when I installed Apache from source. I'm not even curious as to why this is an issue, nor why libgcc_s.so.1 is needed. At this point I'm like “give the system what it wants, not what I want to give it” (which I can't say in polite company). Running configure –help didn't show any obvious means of enabling the use of libgcc_s.so.1, and not wanting to dive deep into a maze of twisty Makefiles all alike, I decided on the next best thing. I went to the Apache build directory, deleted the existing httpd binary and ran make. This gave me the final line used to build the executable:

/home/spc/apps/httpd-2.4.54/srclib/apr/libtool --silent --mode=link gcc -std=gnu99  -g -O2 -pthread -o httpd  modules.lo buildmark.o -export-dynamic server/libmain.la modules/core/libmod_so.la modules/http/libmod_http.la server/mpm/worker/libworker.la os/unix/libos.la -L/usr/local/lib -lpcre    /home/spc/apps/httpd-2.4.54/srclib/apr-util/libaprutil-1.la -lexpat /home/spc/apps/httpd-2.4.54/srclib/apr/libapr-1.la -luuid -lrt -lcrypt -lpthread -ldl

I then reran just this command but with /lib/libgcc_s.so.1 added before the -pthread option. A quick check afterwards:

[spc]brevard:~/apps/httpd-2.4.54>ldd .libs/httpd
	linux-gate.so.1 =>  (0xf77e1000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf77cf000)
	libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0xf77b1000)
	libaprutil-1.so.0 => /usr/local/apache2/lib/libaprutil-1.so.0 (0xf778c000)
	libexpat.so.0 => /usr/lib/libexpat.so.0 (0xf776e000)
	libapr-1.so.0 => /usr/local/apache2/lib/libapr-1.so.0 (0xf7742000)
	libuuid.so.1 => /lib/libuuid.so.1 (0xf773f000)
	librt.so.1 => /lib/tls/librt.so.1 (0xf772a000)
	libcrypt.so.1 => /lib/libcrypt.so.1 (0xf76fc000)
	libpthread.so.0 => /lib/tls/libpthread.so.0 (0xf76ea000)
	libdl.so.2 => /lib/libdl.so.2 (0xf76e6000)
	libc.so.6 => /lib/tls/libc.so.6 (0xf75bb000)
	/lib/ld-linux.so.2 (0xf77e2000)

and it worked—libgcc_s.so.1 is linked in!

I installed the new Apache executable, restarted Apache and … it didn't crash! I then left it to run.

That was a week ago, and so far, so good. The error log is rotated weekly and for the past week no such errors have appeared. Now only if I could nuke from orbit the crawlers sending in silly requests like /cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh—it just clutters up the error log with “invalid URI path” errors.

Sigh.

Saturday, July 27, 2024

Fixing more Apache errors

A week later and I can't say I cleared up all the errors with my web server:

[Sat Jul 27 09:51:45.349454 2024] [cgid:error] [pid 7353:tid 4149003968] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi
[Sat Jul 27 09:51:45.349617 2024] [cgid:error] [pid 7348:tid 3807226800] [client 192.200.113.155:55207] End of script output before headers: boston.cgi
[Sat Jul 27 09:51:45.350209 2024] [cgid:error] [pid 7353:tid 4149003968] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi
[Sat Jul 27 09:51:45.350297 2024] [cgid:error] [pid 7348:tid 3807226800] [client 192.200.113.155:55207] End of script output before headers: boston.cgi
[Sat Jul 27 09:51:45.352660 2024] [cgid:error] [pid 7353:tid 4149003968] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi
[Sat Jul 27 09:51:45.352814 2024] [cgid:error] [pid 7636:tid 3815619504] [client 192.200.113.155:49997] End of script output before headers: boston.cgi
[Sat Jul 27 09:51:45.353377 2024] [cgid:error] [pid 7353:tid 4149003968] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi
[Sat Jul 27 09:51:45.353462 2024] [cgid:error] [pid 7636:tid 3815619504] [client 192.200.113.155:49997] End of script output before headers: boston.cgi
[Sat Jul 27 09:51:45.353790 2024] [cgid:error] [pid 7353:tid 4149003968] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi
[Sat Jul 27 09:51:45.353943 2024] [cgid:error] [pid 7691:tid 3832404912] [client 192.200.113.155:48697] End of script output before headers: boston.cgi
[Sat Jul 27 09:51:45.354685 2024] [cgid:error] [pid 7353:tid 4149003968] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi
[Sat Jul 27 09:51:45.354813 2024] [cgid:error] [pid 7691:tid 3832404912] [client 192.200.113.155:48697] End of script output before headers: boston.cgi
[Sat Jul 27 09:51:45.360184 2024] [cgid:error] [pid 7353:tid 4149003968] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi
[Sat Jul 27 09:51:45.360295 2024] [cgid:error] [pid 7349:tid 3731692464] [client 192.200.113.155:44083] End of script output before headers: boston.cgi
[Sat Jul 27 09:51:45.360856 2024] [cgid:error] [pid 7353:tid 4149003968] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi
[Sat Jul 27 09:51:45.360940 2024] [cgid:error] [pid 7349:tid 3731692464] [client 192.200.113.155:44083] End of script output before headers: boston.cgi
[Sat Jul 27 09:51:45.366567 2024] [cgid:error] [pid 7353:tid 4149003968] (12)Cannot allocate memory: AH01252: couldn't create child process: 12: boston.cgi
[Sat Jul 27 09:51:45.366719 2024] [cgid:error] [pid 7786:tid 3916331952] [client 192.200.113.155:55205] End of script output before headers: boston.cgi

There are more entries like this, but you get the idea. Apache can't run mod_blog for some reason. Checking the access log I can match these up to the following requests:

XXXXXXXXXXXXXXX - - [27/Jul/2024:09:51:45 -0400] "GET /2006/07/16/hsr-carpet-1.jpg HTTP/1.1" 500 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43" -/- (-%)
XXXXXXXXXXXXXXX - - [27/Jul/2024:09:51:45 -0400] "GET /2006/07/15/flapper.jpg HTTP/1.1" 500 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43" -/- (-%)
XXXXXXXXXXXXXXX - - [27/Jul/2024:09:51:45 -0400] "GET /2006/07/17/scrapes.jpg HTTP/1.1" 500 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43" -/- (-%)
XXXXXXXXXXXXXXX - - [27/Jul/2024:09:51:45 -0400] "GET /2006/07/16/hsr-carpet-2.jpg HTTP/1.1" 500 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43" -/- (-%)
XXXXXXXXXXXXXXX - - [27/Jul/2024:09:51:45 -0400] "GET /2006/07/17/luxor.jpg HTTP/1.1" 500 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43" -/- (-%)
XXXXXXXXXXXXXXX - - [27/Jul/2024:09:51:45 -0400] "GET /2006/07/18/rushhour.jpg HTTP/1.1" 500 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43" -/- (-%)
XXXXXXXXXXXXXXX - - [27/Jul/2024:09:51:45 -0400] "GET /2006/07/18/area51.jpg HTTP/1.1" 500 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43" -/- (-%)
XXXXXXXXXXXXXXX - - [27/Jul/2024:09:51:45 -0400] "GET /2006/07/18/littlealeinn.jpg HTTP/1.1" 500 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43" -/- (-%)
XXXXXXXXXXXXXXX - - [27/Jul/2024:09:51:45 -0400] "GET /2006/07/18/quik-pik.jpg HTTP/1.1" 500 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.43" -/- (-%)

To me, this is obviously a crawler, despite claiming to be every possible web browser in existance—is it Windows? MacOS? Linux? Yes. But what's interesting is that all the errors seem related to serving up images.

The way my blog works, all requests to posts are fed through mod_blog, and if said request is for an image, it just copies the file out. It works, but if the server gets slammed just a bit too hard, it breaks down. If only there was some way to get Apache to serve the images directly instead of having to go through mod_blog.

Wait! There is!

I've been using Apache for well over twenty-five years now, so it was a relatively easy issue to solve. First off, point Apache to the directory where all the data for mod_blog is stored.

Alias                 /XXXXX/ /home/spc/web/sites/boston.conman.org/journal/

<Directory /home/spc/web/sites/boston.conman.org/journal>
  Options       None
  AllowOverride None
  <LimitExcept GET HEAD>
    Require valid-user
  </LimitExcept>
</Directory>

The first directive maps the “web directory” /XXXXX/ to the actual directory on the file system. The Directory block restricts what can be viewed and how it can be viewed. All that's left is to throw all requests to images to this directory:

RewriteRule ^([0-9][0-9][0-9][0-9]/[0-9][0-9]/[0-9][0-9]/.*\.(gif|png|jpg|ico)) XXXXX/$1 [L]

What this does is rewrites a request like /2015/07/04/Desk.jpg to /XXXXX/2015/07/04/Desk.jpg, which references the image directly on the file system, letting Apache serve it up directly. This rule goes before the other RewriteRules so Apache serves the image up before mod_blog sees the request.

An easy fix that should lighten the load on Apache as it serves up my blog. I'll see in a week if it all goes to plan.

Sunday, July 28, 2024

Notes on an overheard conversation while pulling into the driveway

“You know, it's not a song unless you can sing the lyrics to it.”

“There are lyrics!”

“Oh yes? Then sing them!”

“Okay, granted—I can barely hear them, but they're there!”

“Well, can you carry this into the house?”

“Sigh. Okay, I will, even though you don't seem to appreciate the finer things in life.”

“It's still not a song.”

“Philistine!”

The case of the well-known location being denied when it doesn't exist

I was checking up on the Apache error log when I noticed the following:

[Sun Jul 28 18:47:21.455848 2024] [authz_core:error] [pid 25597:tid 3916331952] [client 74.173.118.3:53916] AH01630: client denied by server configuration: /usr/local/apache2/htdocs/.well-known
[Sun Jul 28 18:47:59.176743 2024] [authz_core:error] [pid 25598:tid 3916331952] [client 74.173.118.3:53918] AH01630: client denied by server configuration: /usr/local/apache2/htdocs/.well-known/
[Sun Jul 28 18:50:33.324290 2024] [authz_core:error] [pid 25759:tid 3832404912] [client 74.173.118.3:53922] AH01630: client denied by server configuration: /usr/local/apache2/htdocs/.well-known

That's odd, I thought. I don't have that directory in any of my virtual domains, so why is it denied by the server configuration? And thus I fell into a rather odd rabit hole of Apache configuration oddities.

I created the directory. I can see it when I go to https://boston.conman.org/.well-known/. But when I go to http://boston.conman.org/.well-known/ I would get a “403 Forbidden” error, and the above error message logged. The only difference between the two links—one is HTTPS (that works) and the other is HTTP (that fails). But if I go to http://boston.conman.org/ (HTTP—thus insecure), it would redirect to https://boston.conman.org/ (HTTPS—secure). In fact, every link to boston.conman.org via HTTP redirects, except for those starting with /.well-known/.

Huh?

It turns out, this started a a year and a half ago when I enabled the Apache module mod_md and used the MDRequireHttps directive. This directive will cause plain HTTP requests to be redirected to HTTPS, because I know, I just know, that one day Google is going to take HTTP out behind the shed and then no one will be able to use plain HTTP anymore because The All Knowing Google knows whats best for us (All Praise Google, Peace Be Upon It!) so I might as well get in front of that before it happens. But there's a small bit in the documentation:

MDRequireHttps Directive

…

You can achieve the same with mod_alias and some Redirect configuration, basically. If you do it yourself, please make sure to exclude the paths /.well-known/* from your redirection, otherwise mod_md might have trouble signing on new certificates [emphasis added].

Okay, so that explains why http://boston.conman.org/.well-known/ (HTTP—insecure) isn't being redirected—it's a side effect from mod_md. But that doesn't explain the error where it's denied by the server configuration. A bit more digging, and I find the following in the Apache configuration:

DocumentRoot "/usr/local/apache2/htdocs"
<Directory "/usr/local/apache2/htdocs">
    #
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.apache.org/docs/2.4/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks

    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   AllowOverride FileInfo AuthConfig Limit
    #
    AllowOverride None

    #
    # Controls who can get stuff from this server.
    #
    #Require all granted
    Require all denied
</Directory>

Aha! When I set things up, I configued the HTTP site with:

<VirtualHost 71.19.142.20:80>
	ServerName	boston.conman.org
	Protocols	h2 h2c http/1.1 acme-tls/1
</VirtualHost>

I don't with a directory because I know that all requests will be redirected anyway, so why bother? Only in this case, mod_md isn't doing a redirect for /.well-known/ and because there's no <Directory> directive, the requests fall back to the Apache default web directory, which is configured to be unreadable for every request.

Wow!

A quick change to make the default web directory available and no more AH01630 errors. So now I wait to see if this breaks anything. Wheeeee!

Wednesday, July 31, 2024

Extreme metal lawn sculptures, Boca Raton edition

[A well manicured lawn next to the street, with a collection of metal rods all askew forming some form of structure] It kind of, sort of, looks like a headless version of the Burning Man sculpture. Or perhaps a loose interpretation of a Feynman Diagram. Hard to say …

“What is that?”

“I … don't know. It looks like some form of metal sculpture.”

“I didn't think [our neighbor] was into metal sculptures.”

“Maybe he picked up a new hobby.”

“It could also be a bunch of rusted out and busted lawn chairs.”

“It's so hard to tell with modern art.”

Obligatory Picture

Trying to get into the festive mood this year

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

Obligatory AI Disclaimer

No AI was used in the making of this site, unless otherwise noted.

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site: https://boston.conman.org/, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

https://boston.conman.org/2000/08/01

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.