The Boston Diaries

Saturday, January 01, 2022

Ramming in the New Year

HAPPY NEW YEAR!

Janurary 1^st is the other fireworks happy day, but this year it's been relatively quiet. There were some fireworks, but there were far off in the distance and not across the street.

There was, however, something new—a neighbor (and not the one who usually does fireworks) bringing in the New Year with a ram's horn.

It's back to the saltmine (and in this case, “saltmine” is the name of the Corporate issued laptop, not to be confused with the Corporate Overlords' managed laptop, named “Satan”). I check my email only to find half a dozen emails from last week (which nearly everyone at The Corporation had off, including me) saying my password for the Corporate network was expiring and I should change it. I also found half a dozen emails from last week (which nearly everyone at The Overlords' Corporation had off, including me) saying my password for the Corporate Overlords' network was expiring and I should change it (yes, there are two different networks for hysterical reasons). And of course, the two different networks have different password rotation lengths that are timed such that they both expire during vacations. And yet, no matter how many times I point out NIST Special Publication 800-63b, section 5.1.1.2, which states: “Verifiers SHOULD NOT require memorized secrets to be changed arbitrarily (e.g., periodically),” these stupid password expirations keep happening. I guess I'll have to wait for another few CSOs to rotate through the office before we can finally stop the “Password Changing Dance.”

Sigh.

“What idiot did this? Oh … said idiot is me.”

Not only is “Project: Bradenburg” being built from git, but it seems operations has finally gotten “Project: Seymore” build servers working with git. I decided to check it out and … what the … ? What did they do to the code? Submodules? Outrageous!

I poked a bit deeper, and it turns out, yeah, that idiot was me! I had completely forgotten the eldritch horrors I unleashed nearly two years ago. Oh, is 2022 going to be a fun year …

Tuesday, January 04, 2022

Fixing bugs by cleaning code

I figured out an approach to “Project: Bradenburg.” I'm dumping the C++ code as it was never used in production and as someone who is more confortable with C than C++, I think that's the best choice right now. To that end, I've made a checklist of the items that need addressing right now—and they're all related to cleaning up the code.

I've normalized the whitespace in all the source files. Tab characters were introduced by the previous developer, who used a different tab stop setting than I'm used to, so the code formatting is just way off. I removed the #pragma declarations and fixed all the resulting warnings. And for the warnings I don't necessarily agree with, like -Wformat-truncation, I can supress in the makefile (well, GNUmakefile if you want to be pedantic):

src/app/stormgr.o : override CFLAGS += -Wno-format-truncation

This disables the warning

src/app/stormgr.c:194:3: note: ‘snprintf’ output between 2 and 4097 bytes into a destination of size 4096

from code like this:

snprintf(filename, sizeof(filename),"%s/", GetGlobalConfig()->spool_dir);

It's nice I get the warning, but in this case, if we get a filename of 4097 characters long we have other issues to worry about.

And that's pretty much the only warning I have to supress. The other warnings were all issues that needed to be addressed, and in one case, an actual bug was fixed.

Right now I'm cleaning up what is, to me, a pointless abstraction dealing with syslog(). The previous developer wrote a wrapper around syslog() called SysLog(). It would be one thing if there was some reason to wrap syslog(), like we would be running on Windows (we won't) or needed to redirect output to a file (we don't). But no, all the wrapper does is:

void SysLog(int priority, const char *fmt, ...)
{
  va_list marker;
    
  if (fSyslogOpened)
  {
    va_start(marker, fmt);
    vsyslog(priority, fmt, marker);
    va_end(marker);
  }
}

And the funny thing—syslog() already handles the case when openlog() hasn't been called. So there's no reason for this wrapper whatsoever in this code base. What makes this even more special is how the developer called SysLog():

SysLog(LOG_NOTICE, BRADENBURG5024,  dbuf);

And elsewere in the code in some header file:

#define BRADENBURG5024    "BRADENBURG5024: OUTBIND connection accepted from %s"

I … um … I … erm … yeah. I can make sense of this, in the “we might want to translate the log messages to another language” argument. But we don't sell this product—it's for internal use. And there are other ways to go about doing this rather than separate the format string from its use. Nothing like finding nine instances of this:

SysLog(LOG_EMERG, BRADENBURG0001);

where BRADENBURG0001 is defined as:

#define BRADENBURG0001    "BRADENBURG0001: %s"

For those unwise in the ways of C programming, this is calling a function with effectively a missing parameter and the compiler can't warn about it because the format string (which informs the called function about what parameters to expect) exists in a different file.

So just by cleaning up the code and removing pointless abstractions I'm fixing bugs.

Wednesday, January 05, 2022

Meeting my new manager before training my new manager

I finally met my new manager! It's been … what? 3½ months? … since it was announced. I decided to ask a VP of the Corporate Overlords who was my actual manager, M1 (who was promoted) or M2 (who is to replace the promoted manager). The VP said M2, and that since I have yet to meet him, I should invite him to the next department meeting. Why it should be up to me to invite M2 to our daily meeting and not M1 is apparently beyond my pay grade, but I invited him.

And it turns out the situation is still … complicated. M1 is still my “manager” for some things for the forseeable future, and M2 is my “manager” for the other things not handled by M1 for the forseeable future. But overall, the meeting went well. We shall see how things go.

Friday, January 07, 2022

“You can't fix the bug until you have filled out form 27B/6.”

I swear, I just cannot adjust to the new development process. Last month, as I was starting work on “Project: Bradenburg,” I ended up using GCC 11, and just on a whim, I decided to try GCC 11 on “Project: Lumbergh” and that's when I found an issue—“Project: Lumbergh” crashed immediately. It took about ten minutes to track down the issue—undefined behavior! Effectively, “Project: Lumbergh” was taking a pointer to a variable in an inner scope and using it outside said scope (don't worry, there's an example of what I'm talking below). I duly recorded a bug in Jira, but did not check in the patch as the other developer, CZ, was doing a bunch of work on “Project: Lumbergh” and I wanted to run the bug by him first just so he knew about it. In retrospect, I think I should have just checked in the patch and asked forgiveness, because the ensuing “process” nearly killed me.

First off, the bug basically languished due to the holliday season and everybody taking vacation and what not. When I brought up the bug on Monday, CZ demanded a test case to show it failing. Problem—there was no reason to create a specific test case, because any test would have shown the crash. Second problem—it's undefined behavior—it only shows (as far as I know) when compiled with GCC 11. We don't use GCC 11 in production, and the GCC version we use in production produces code that runs, thus not finding the bug any ealier.

CZ tried using valgrind to locate the issue and couldn't. Of course CZ couldn't—the executable worked. As I tried to state, it only fails when compiled with GCC 11 (which we as a company don't use yet).

I spent the week trying to convince him to just accept the patch because it removed undefined behavior. CZ was adamant in trying to reproduce the issue for himself. Because with our Corporate Overlords, all bugs must be reproducable, must have a test case, and must pass said test case, before it will be patched.

Even in the case of blatant undefined behavior!

ARE YOU XXXXXXX KIDDING ME? IS OUR CODE SO XXXXXXX FRAGILE THAT MOVING THREE VARIABLES DECLARATIONS WILL XXXXXXX BREAK THE PROGRAM? IS THAT WHAT YOU ARE SO XXXXXXX AFRAID OF? XXXXXXXXXXXXXXXXXXXX!

[Calm down! Breath! In … out … in … out. —Editor]

Sigh.

Eventually, I was given the go-ahead to submit the patch. Here was my revision notes:

Bug fix XXXXXX—remove undefined behavior

This was found using GCC 11, and it's very hard to reproduce, since the code invoked undefined behavior by storing pointers to data in a scope that technically no longer exists when it's referenced.

The reason this is hard to find depends upon how the compiler lays out the stack frame for the given function. It can, for instance, collect all the variable created at all the scopes in the function and set aside space for all of them at once, which in that case the code will work without issue. Or it could collect all the variables from all the scopes, and create just enough space for all the variables, allowing the variables from one scope to reuse memory for another scope, in which case, it may work, or not, depending upon intervening scopes. Or, the compiler can create the scoped variables when the code enters the scope, in which case the code may lead to undeterministic behavior, depending upon the code following the scope.

An example:
struct bar
{
  char b[MAX];
};

struct foo
{
  char        f[MAX];
  struct bar *sub;
};

static int foo(A a, B b, C c)
{
  struct foo f;	  
  bool       flag;
  
  flag = to_display(f.f,sizeof(f.f),a);
  if (flag)
  {
    struct bar b;
    
    b.b = to_other_display(b.b,sizeof(b.b),b,c);
    f.sub = &b;
  }
  else
    f.sub = NULL;
    
  if (check_this())
  {
    int x = tointeger(b);
    char buf[MAX];
    
    to_c_display(buf,sizeof(buf),c); 
    syslog(LOG_DEBUG,"check=%d c=%s",x,buf);
  }
    
  do_something(&f);
  return 0;
}
In this sample function, compiler A may collect all the local variables into a single block at the start of the function:
struct foo f;
bool       flag;
struct bar b;
int        x;
char       buf[MAX];
and even though this provokes undefined behavior, it will still work, and valgrind won't find an issue. Compiler B might create space for all the variables, but could reuse the space for the two sub blocks:
struct foo f;
bool       flag;
union
{
  struct bar b;
  struct { int x ; char buf[MAX]; }
};
(not valid C syntax, but it should give the intent I'm going for). In this case, the space pointed to by b could be overwritten if check_this() returns true. Compiler C may only create the space as needed, so the resulting assembly code may look something like:
foo	
	push	rbp
	mov	rbp,rsp
	sub	rsp,sizeof(struct foo) + sizeof(flag) + padding
	…
	cmp	al,[rbp+flag]
	jz	foo_skip

	sub	rsp,sizeof(struct bar) + padding
	…
	lea	rbx,[rbp - f];
	lea	rdx,[rbp - b];
	mov	[rbx + sub],rdx
	…
	add	rsp,sizeof(struct bar) + padding

foo_skip:
	…
	call	check_this
	tst	al
	jz	foo_skip2

	sub	rsp,sizeof(int) + sizeof(buf) + padding
	…
	add	rsp,sizeof(int) + sizeof(buf) + padding

foo_skip2:

	call	do_something
And in this case, if check_this() even returns false, data in struct bar could be overwritten just with the call to check_this() itself.

This is the very definition of “undefined behavior” and in this case, it's might have been hard to find, even with tools like valgrind, since it really depends upon the code generated by the compiler. I think we're lucky in that GCC 11 layed out the code such that it crashed and thus, I was able to isolate the issue long before it becomes important.

Tuesday, January 11, 2022

Let's look at some bots that aren't the MJ12Bot

I think it's time I stop blogging about work after my previous post. Work is getting a tad too depressing to think about and my cynical side is saying that it won't matter where I go, it'd be more or less the same with a higher probability of forced Microsoft Windows use. So instead of that depressing topic, let's take a look at something much lighter and less depressing—the current state of Internet robots crawling my various sites!

Two weeks later and there are still bots attempting to follow endless redirections. I thought maybe I could attempt to figure out a contact, but alas, they're coming from all over the place (and yes, I'm finally naming IP addresses):

Top 20 Gemini based bots caught in redirection Hell
IP address	# requests
18.134.208.136	933
18.132.248.127	850
3.8.92.131	817
18.169.194.52	745
3.8.210.87	728
13.40.97.54	715
18.170.56.106	713
3.8.134.65	682
35.176.22.93	681
18.130.231.183	681
13.40.67.85	667
13.40.137.233	666
18.132.46.166	659
3.8.24.209	641
18.170.107.207	637
13.40.155.157	634
35.178.170.215	577
18.130.216.34	573
13.40.145.207	572
35.179.76.79	564

They're all pretty much from Amazon Web Services so who knows who is running these bots. Just blocking them is too easy a solution—at this point, I'd like to do something to get their attention (as if thousands of links they are crawling are suddenly listed as “gone” isn't enough of a clue). I don't necessarily mind bots crawling my sites, unless they're doing stupid things. I shall have to think on this a bit more.

I also had high hopes that I could stop empty requests to my Gemini server (which isn't allowed at all by the specification) by returning a non-standard response code with the text “Not a gopher server” but alas, that is still happening. Does nobody bother checking results of their bots running? I guess not.

And speaking of gopher, it's better there than Gemini. Yes, there are a few agents that are attempting to use TLS, but fortunately, they cache previous failures so it's not every request. There are a few bots out there trying to exploit RDP (not much I can do about those) and a few that are confused into thinking my gopher site is actually my Gemini site sans TLS (What?). But I can live with 155 failed gopher requests out of 10,423 over the past month.

And while I'm checking bots, I can't forget the web crawlers. And not much has changed on that front since July 2019 except that MJ12Bot has kept their promise never to crawl my site again. The Knowledge AI (which I cannot find any information on) is still the number one agent, with 68,000 requests in Debtember 2021, followed by 21,000 requests from Amazonbot. And it seems that the bots in general are making fewer requests to non-existant pages (I mean, back in June 2019, The Knowledge AI made 170 bad requests; last month, 1).

So, with the exception of bots stuck in redirection Hell in Gemini, things on the crawler front are looking pretty good.

Sunday, January 16, 2022

A most persistent spam, part VI

It seems that “Aleksandr” may have changed his name to “Mayboroda,” but it looks like it's the same type of weird spam I've since blocked successfully. Only here, reader Roberto found a way to block the spam for users of Postfix (and I did get Roberto's permission to post this email):

From
Robysampler <XXXXXXXXXXXXXXXXXXXXX>

To
sean@conman.org

Subject
About "Mayboroda_aleks" on your personal blog

Date
Sun, 16 Jan 2022 23:04:07 +0100

Dear Mr. Sean

My name is Roberto from Italy.

i've read your personal blog about the mayboroda aleks spammer, who's bothering me, filling my own company email since one and half years, at least.

as you figured out "Mayboroda", keeps changing IPs and domain/subdomains to evade every try to block him.

luckly, my company mail is served by a linux machine i own, so i have direct access to it, and as final solution i've choose to do some fine tuning in postfix config.

i've add inside postfix "main.cf" file:
smtpd_recipient_restrictions = check_sender_access regexp:/etc/postfix/rejected.senders
then i've add in "rejected.senders":
/s[0-9]{1,2}.[a-z]*.ru/ REJECT
/info@.[a-z]*.ru/ REJECT
in this case you'll provide to your postfix daemon, some rejecting rules based on regular expressions.

based on hundreds of mails "Mayboroda" has sent me, i figured out the main pattern for his emails usually are

info@randomdomain.ru

or

something@s(1 or 2 numbers).randomdomain.ru

after setting up your postfix you can check out the result using the command
postmap -q "your test email here" regexp:/etc/postfix/rejected.senders
for example
postmap -q "info@s4.mayboroda.ru" regexp:/etc/postfix/rejected.senders
the shell returns REJECT

this will works until "Mayboroda" will continue to use the same pattern in the mail sender

I hope you'll appreciate my advices.

have a nice day and happy new year

Roberto

Best Regards

I do appreciate your advice, Roberto. Thank you. I'm sure other people will find this useful as well.

Monday, January 17, 2022

A most persistent spam, part VII

I received a follow-up message from Rooberto about the “Aleksandr Russian spam emails:

Hi Sean.

Thanks very much for your fast reply.

i have some good news about "Mayboroda"

here some lines of my postfix log showing "Mayboroda" has tryed again, sending me some spam today:

Jan 17 11:48:47 mydomain postfix/smtpd[23894]: warning: hostname tefalongo.ru does not resolve to address 185.186.3.10
Jan 17 11:48:47 mydomain postfix/smtpd[23894]: NOQUEUE: reject: RCPT from unknown[185.186.3.10]: 450 4.7.25 Client host rejected: cannot find your hostname, [185.186.3.10]; from=<info@s7.kroshem.ru> to=<booking@mydomain.net> proto=ESMTP helo=<s7.kroshem.ru>
Jan 17 12:18:49 mydomain postfix/smtpd[24258]: warning: hostname tefalongo.ru does not resolve to address 185.186.3.10
Jan 17 12:18:49 mydomain postfix/smtpd[24258]: NOQUEUE: reject: RCPT from unknown[185.186.3.10]: 450 4.7.25 Client host rejected: cannot find your hostname, [185.186.3.10]; from=<info@s7.kroshem.ru> to=<info@mydomain.net> proto=ESMTP helo=<s7.kroshem.ru>
Jan 17 12:18:49 mydomain postfix/smtpd[24258]: NOQUEUE: reject: RCPT from unknown[185.186.3.10]: 450 4.7.25 Client host rejected: cannot find your hostname, [185.186.3.10]; from=<info@s7.kroshem.ru> to=<booking@mydomain.net> proto=ESMTP helo=<s7.kroshem.ru>
Jan 17 12:48:49 mydomain postfix/smtpd[24629]: connect from s7.kroshem.ru[185.186.3.10]
Jan 17 12:48:49 mydomain postfix/smtpd[24629]: NOQUEUE: reject: RCPT from s7.kroshem.ru[185.186.3.10]: 554 5.7.1 <info@s7.kroshem.ru>: Sender address rejected: Access denied; from=<info@s7.kroshem.ru> to=<info@mydomain.net> proto=ESMTP helo=<s7.kroshem.ru>

in particular the last line shows that the regular expression has found a match on "info@s7.kroshem.ru" and replyed "Sender address rejected: Access denied" and REJECTED the incoming Email.

there are some other tweaks you can implement into your "main.cf" postfix configuration file that will help you to avoid junk emails

the following is a partial extract from my postfix "main.cf" configuration:

smtpd_recipient_restrictions = permit_mynetworks,
	permit_sasl_authenticated,
	check_sender_access regexp:/etc/postfix/rejected.senders, #check recipients by regular expression
	check_policy_service unix:private/policyd-spf,
	reject_rhsbl_helo dbl.spamhaus.org,           #check if domain or ip is flagged as spam in spamhouse database
	reject_rhsbl_reverse_client dbl.spamhaus.org, #check if domain or ip is flagged as spam in spamhouse database
	reject_rhsbl_sender dbl.spamhaus.org,         #check if domain or ip is flagged as spam in spamhouse database
	reject_rbl_client zen.spamhaus.org            #check if domain or ip is flagged as spam in spamhouse database

smtpd_sender_restrictions =  permit_mynetworks,
	permit_sasl_authenticated,
	reject_unknown_reverse_client_hostname, #Reject the request when the client IP address has no address->name mapping.
	reject_unknown_client_hostname,         #Reject the request when 1) the client IP address->name mapping fails, or 
                                                #2) the name->address mapping fails, or 
                                                #3) the name->address mapping does not match the client IP address.
	reject_unknown_sender_domain            #Reject the request when Postfix is not the final destination for the sender address

Many of these tweaks i've implemented were taken from the document at the following webpage:

http://www.armellin.com/friends/postfix/postconf.5.html

Feel free to publish our conversation in your blog as you wish.

It's nice to help other people to get rid of the plague of "Mayboroda" :D

Thanks Sean

Best Regards

Roberto

Thank you again, Roberto.

Tuesday, January 18, 2022

Extreme tourism, Cashiers, North Carolina edition

The Marginalia Search Engine is such a cool search engine. It's like visiting the web in the 90s with the quirky personal sites with ugly designs and no advertising. Try out some random sites.

Very cool.

While there, I decided to see what results it had for Brevard, NC and boy, some of the results are incredible. I did not know that Cashiers, North Carolina is a hotbed of UFO activity. Had I know, I might have tried looking for an alien license plate at (what is now) the Cashiers Valley Smokehouse. Although, had we delayed our 2015 visit to Brevard to mid-November, we might have seen an actual UFO!

Darn our luck!

I'm not saying it's aliens, but … it's aliens

Right after finding out about Cashiers being a hotbed of UFO activity, my keyboard flaked out on me. The “o,” and “r” keys would register sporadically, which is odd given that I only use IBM Model M keyboards. And why then? Is it something they don't want me to know? Or maybe … aliens? Very odd.

I've probably been using that keyboard for nearly twenty years (it probably just needs good cleaning, but it requires a special screw driver to open), but fortunately, I have an entire stash of IBM Model M keyboards at hand to swap out. so I think I'm set for the rest of my life Because have you seen the prices for IBM Model M keyboards? Averaging around $150, with spikes up to $650! Insane! I never paid more than $10 for any of mine.

I pulled another keyboard from my stash, gave it a light dusting, and I'm good to go.

I'm still amazed this isn't Stevie Wonder

Today I learned that the song “Virtual Insanity” is not by Stevie Wonder, but Jamiroquai. The video is also quite insane, what with Jamiroquai sliding around the room, along with the furniture—trippy stuff, and something I had to find out how it was done. And like all magic tricks, it's both insanely simple and yet, amazing how well it's pulled off.

Wednesday, January 19, 2022

“The master then called out to three senior monks, to attend the example of the bridge builder, and to hear of the discipline of a true engineer.”

Surfing the web, I came across “The Codeless Code: Case 154 A Bridge To Nowhere”. It hit pretty hard, especially given the situation at The Corporation, despite repeated exhortions about how well things worked in the past and how not so well things are working now, they aren't going to change course—the process of the process is the process and all that.

It was the final paragraph of “A Bridge to Nowhere” where I enlightened, and laughed out loud. We'll see how long that lasts.

Thursday, January 20, 2022

Notes on an overheard conversation at a medical lab

“Which arm?”

“It's a pain either way.”

“Okay, we'll do this one.”

“Are you attempting to amputate my arm?”

“Sigh. No. Please make a fist.”

“Aaaaaaaaaaah!”

“That was just the achohol wipe, sir.”

“Are you sure?”

“This is the needle.”

“Gaaaahhhhhhhhhhhhhhhh!”

“Hmm … ”

“Aaaaaaaaarrrrrrrrraaaaahhhhhhhhhh!”

“I think I'll have to try the other arm.”

“I'm dying here!”

“What did I do to deserve this?”

“Aaaaaaaaaaaaaaaaaaaaaah!”