The Boston Diaries

Monday, August 06, 2018

I guess we'll see how well CBOR works

The Corporate Overlords of The Corporation have spoken, and lo, they said “Worry not, for you do not have to deal with the stupidity—we shall deal.” So great, we don't have to drink the REST/HTTPS über alles Kool-Aid™, but instead query our Corporate Overlords for the data, who have drunk the REST/HTTPS über alles Kool-Aid™ (or were forced to by the company we query for the data—the end effect is the same though).

Sigh.

So for now, we still make our queries via UDP, only now in CBOR—the legacy format of DNS is apparently too arcane to support any more.

I have a Lua module but we also need one in C. There are some existing ones, but they have their issues (either an alien build system or missing some critical CBOR feature) so I've been working on a C library I started a few years ago and never finished.

It's working now, and to test it, I've been using valgrind to ensure memory safety, in addition to using /dev/urandom to generate random garbage:

GenericUnixPrompt% dd if=/dev/urandom count=2 | tee bad-data | valgrind ./testdecoder

dd is one of those relatively arcane Unix programs that I find useful on occasion (like here to generate some random data). tee I use to record the data so I can play it back when valgrind finds an issue. This is a reasonable way to fuzz a program. It did find several issues that could have lead to a crash, and I don't leak any memory so the code should be good to go.

It seems that checking the TLS API version number is useless

I've pretty much finished the Lua TLS module and before releasing it, I thought it might be nice to ensure it compiles with previous versions of libtls. The main header file contains the defined value TLS_API, which I assume is updated whenever the API is updated. So I began the arduous procedure of downloading previous versions of libtls to ensure I can compile against any version.

I started with LibreSSL version 2.7.4 (current when I started—they are now up to 2.8.0 as I write this). The defined value TLS_API had a value of “20180210”. I checked version 2.7.0 and no change in libtls. It wasn't until I got into the pre-2.7 versions that things started going south.

The previous version of TLS_API, “20170126”, was first defined in 2.5.1, and last used in 2.6.5. But the API changed quite a bit between versions 2.5.1 and 2.6.5. Five functions were added:

tls_config_set_crl_file()
tls_config_set_crl_mem()
tls_config_set_ecdhecurves()
tls_peer_cert_chain_pem()
tls_unload_file()

What's the point of having a defined value like TLS_API if it doesn't change when you add new functions?

Fortunately, the defined value LIBRESSL_VERSION_NUMBER is updated per version, so at least I can use that.

Sigh.

Tuesday, August 07, 2018

All I'm asking for is some consistency between APIs and version numbers

When I first started working with libtls, I thought that TLS_API designated a change in API so that one could work with different versions of the library without breaking the compilation process. Sadly, that wasn't the case, so I switched to using LIBRESSL_VERSION_NUMBER, as that seemed to regularly change with each version.

I was doing this so that my Lua wrapper could be compiled with any version of libtls. Why break things unnecessarily? And things were going fine until I hit version 2.2.1, and well …

Mismatches in `libtls` between functions added, `LIBRESSL_VERSION_NUMBER` and `TLS_API`
Version	Functions	`LIBRESSL_VERSION_NUMBER`	`TLS_API`
Version	Functions	`LIBRESSL_VERSION_NUMBER`	`TLS_API`
2.1.2	`tls_accept_socket()` `tls_client()` `tls_close()` `tls_config_clear_keys()` `tls_config_insecure_noverifycert()` `tls_config_insecure_noverifyhost()` `tls_config_set_ca_file()` `tls_config_set_ca_path()` `tls_config_set_cert_file()` `tls_config_set_cert_mem()` `tls_config_set_ciphers()` `tls_config_set_ecdhcurve()` `tls_config_set_key_file()` `tls_config_set_key_mem()` `tls_config_set_protocols()` `tls_config_set_verify_depth()` `tls_config_verify()` `tls_configure()` `tls_connect()` `tls_connect_fds()` `tls_connect_socket()` `tls_error()` `tls_free()` `tls_init()` `tls_read()` `tls_reset()` `tls_server()` `tls_write()`	0x20000000	20141031
2.1.4	`tls_load_file()`	0x20000000	20141031
2.2.0	`tls_accept_fds()`	0x20000000	20141031
2.3.0	`tls_config_insecure_noverifytime()` `tls_config_prefer_ciphers_client()` `tls_config_prefer_ciphers_server()` `tls_config_verify_client()` `tls_config_verify_client_optional()` `tls_conn_cipher()` `tls_conn_version()` `tls_handshake()` `tls_peer_cert_contains_name()` `tls_peer_cert_hash()` `tls_peer_cert_issuer()` `tls_peer_cert_provided()` `tls_peer_cert_subject()` `tls_read()` (paramter change) `tls_write()` (parameter change)	0x20030000	20141031
2.3.1	`tls_peer_cert_notafter()` `tls_peer_cert_notbefore()`	0x20030001	20141031
2.4.0	`tls_config_keypair_file()` `tls_config_keypair_mem()`	0x2040000f	20141031
2.5.0	`tls_accept_cbs()` `tls_config_add_keypair_file()` `tls_config_add_keypair_mem()` `tls_config_alpn()` `tls_conn_alpn_selected()` `tls_conn_servername()` `tls_connect_cbs()`	0x2050000f	20160904
2.5.1	`tls_ocsp_process_response()` `tls_peer_ocsp_cert_status()` `tls_peer_ocsp_this_update()` `tls_peer_ocsp_url()` `tls_config_add_keypair_ocsp_file()` `tls_config_add_keypair_ocsp_mem()` `tls_config_add_ticket_key()` `tls_config_keypair_ocsp_file()` `tls_config_keypair_ocsp_mem()` `tls_config_ocsp_require_stapling()` `tls_config_ocsp_staple_file()` `tls_config_ocsp_staple_mem()` `tls_config_session_id()` `tls_config_session_lifetime()` `tls_peer_ocsp_crl_reason()` `tls_peer_ocsp_next_udpate()` `tls_peer_ocsp_response_status()` `tls_peer_ocsp_revocation_time()`	0x2050100f	20170126
2.6.0	`tls_config_crl_file()` `tls_config_crl_mem()` `tls_peer_cert_chain_pem()` `tls_unload_file()`	0x2060000f	20170126
2.6.1	`tls_config_echdecurves()`	0x2060100f	20170126
2.7.0	`tls_config_session_fd()` `tls_conn_session_resumed()`	0x2070000f	20180210

I'm not asking for much. I'm not asking for slavish adherance to semantic versioning. I'm just asking for a consistent way to check an API to I can support earlier versions of a library.

Don't get me wrong, I'm glad that libtls exists, and as an API, it's much nicer than the eldritch horror of OpenSSL.

I just wish they had updated TLS_API (or LIBRESSL_VERSION_NUMBER) consistently. Otherwise, why have them in the first place?

Oh wait … I see now … send a completely crazy email so I'll post it

I get quite a bit of spam asking to add links to old entries. It's clear that they do a search for something like “Lost Wages” and spam the XXXX out of people who link to any page dealing with their search term. But the one I received yesterday …

From

Adam Conrad <adam@shihtzuexpert.com>

To

sean@conman.org

Subject

Article Contribution on Boston.Conman.Org

Date

Mon, 6 Aug 2018 23:38:14 -0700

Sean,

I really like the post on your site at http://boston.conman.org/2007/12/27.1

I’ve also, been writing about Shih Tzu and other small, toy or medium breed dogs and related issues like health, food etc at

The Shih Tzu Expert

http://shihtzuexpert.com/canine-distemper/

I would love the chance to write a unique and an interesting guest post for your blog.

I can write about any topic related to pets.

Our articles are 700+ words, written by native English speakers/writers.

Our content is unique, fresh, detailed, and thoroughly researched.

If it covers medical issues or advice, we always require our in house veterinarian to check accuracy of facts before publication.

A selection of titles we are proposing

What to look for when buying dog food for Shih Tzu and other toy breeds

Grooming advice, How to groom, best hair cuts, best grooming products.

Veterinary approved articles about Health issues, Distemper, Parvo, Obesity, Lack of appetite etc

If there is any topic you want us to cover, please feel free to let us know.

Please feel free to contact me if you have any questions.

Best regards,

Adam Conrad

ShihTzuExpert.com

Seriously … WTF?

The page Adam linked to?

Nothing to do with dogs!

I have entries where I talk about dogs. And cats. And dogs and cats. It's not like Adam lacked entries to choose from.

But nooo! Adam choose one sans dogs.

What?

Notes on an overheard conversation in a typical company lunch room

“Nice music.”

“It's classical.”

“Classical? That's modern jazz.”

“It's Gershwin. He's white, and he's dead. Therefore, it's classical.”

“So is Kurt Cobain, but I don't consider his music classical.”

Friday, August 10, 2018

Just a small observation about an email I received at work

I'm checking the work email and lo, there's email from our Vice President of Product Management requesting us, unironically, to wear our Corporate flair.

Tuesday, August 21, 2018

Not anyone should be allowed to run a debugger

I'm working on “Project: America-On-Line-Instant-Messenger” (hey, don't judge me—it has nothing to to with AOL, it's more a pun on the project name than anything else). I've got a major portion of it running, but it's exhibiting some odd behavior—it runs fine on Linux, but fails rather quickly on Mac OS-X with “not enough memory.”

Problem one—I'm used to using gdb, but Mac OS-X uses lldb. Sigh. I did find a type of Rosetta Stone that shows similar commands in both gdb and lldb, so that issue is solved.

I was then able to track down the message to a source file in LPeg, but when I set a breakpoint on the line in question, it never got triggered. Puzzling, but I was not able to solve the issue by the time I had to leave work.

So now I'm home and I have an idea to scan more than just the .c files. It's a simple matter to log into my Mac workstation at work to see if that's the issue. I start looking, and lo, I find the message in two more locations, hidden behind a #define in a header file. I narrow it down to just one location (the other location pre-loading the string into the Lua VM).

First, the lldb commands are bit more verbose than those of gdb. Second, no matter what I do, all I get is:

[spc]XXXXXXXXXXXXXXXXXXXXXXXXX>lldb XXXXXXXXX
(lldb) target create "XXXXXXXXX"
Current executable set to 'XXXXXXXXX' (x86_64).
(lldb) process launch -- config.lua
error: process exited with status -1 (lost connection)
(lldb)

Nothing I do works.

Then I remember—when I first ran lldb at the office, this modal dialog box popped up asking for my password in order to debug a program I wrote, running under my account. My initial reaction is, Smooth going, Apple! Way to protect me from myself! But then I realize this is probably to prevent rogue programs trying to attach to running programs to do nefarious things without my knowledge.

It's bad enough I have to jump through hoops to get the computer to run my own written programs. Now this?

XXXX!

It really does seem at times as if the general purpose computer is a dinosaur looking up at the sky, wondering, “What is that?”

Wednesday, August 22, 2018

I don't need no steeeenking debugger

I was still having troubles with lldb accepting breakpoints; the breakpoints were just not being hit. Thinking about the issue, it seemed to me that the code might be requesting a too large amount of memory and an easy eay to do that is to track all allocations made by Lua. It was easy enough to test:

static void *l_alloc(void *ud,void *ptr,size_t osize,size_t nsize)
{
  (void)ud;

  fprintf(stderr,"osize=%zu nsize=%zu\n",osize,nsize);

  if (nsize == 0)
  {
    free(ptr);
    return NULL;
  }
  else
    return realloc(ptr,nsize);
}

int main(int argc,char *argv[])
{
  /* ... */

  L = lua_newstate(l_alloc,NULL);

  /* ... */
  return 0;
}

Then it was a simple matter of running the program and check each allocation:

 ...
osize=0 nsize=64
osize=0 nsize=40
osize=0 nsize=64
osize=0 nsize=0
osize=0 nsize=64
osize=0 nsize=16
osize=0 nsize=64
osize=0 nsize=0
osize=0 nsize=160
osize=0 nsize=148
osize=0 nsize=64
osize=0 nsize=16
osize=16 nsize=32
osize=0 nsize=16
osize=0 nsize=64
osize=0 nsize=18446744073709551600
 ...

Hmmmmm …

Okay, force a core dump we can examine:

static void *l_alloc(void *ud,void *ptr,size_t osize,size_t nsize)
{
  (void)ud;
  (void)osize;

  if (nsize > 10uL * 1024uL * 1024uL)
    abort(); /* dump core! */

  if (nsize == 0)
  {
    free(ptr);
    return NULL;
  }
  else
    return realloc(ptr,nsize);
}

And the problem is immedately resolved.

The Linux version used epoll() for events, while for whatever reason I don't remember, the Mac OS-X version used select() [Yes, I know that's a Linux reference and not a Mac OS-X reference, but the call is the same between the two systems, and I couldn't find a version of the Mac OS-X page online. —Sean], and that code path was … a bit problematic.

The fix was easy—use poll() for events. With that change, the code worked fine on Mac OS-X (not that we use Mac OS-X in production, but it makes it easy to test in development if it works on the Mac).

Thursday, August 23, 2018

The sad thing is, it's not so bad it's amusingly bad, it's just bad

JC, a fellow cow-orker, casually said he was going to throw out an Android tablet as it was garbage. It worked, but apparently was so bad that it wasn't worth keeping. I was curious. So I asked him if I could have it instead.

Wow.

I think I should ask for my money back.

How much of the badness is due to Android and how much of it is due to this being the Mi Traveler^10Q8B I don't know. I think I'm willing to give Android the benefit of a doubt, seeing how it can be modified by the OEM to fit their needs, so I'm attributing all the badness to the Mi Traveler^10Q8B.

And it's baaaaaaaaaaad!

Weight-wise, it's about the same as my iPad, but boy does it feel cheap. The screen is cheap plastic. The case is cheap plastic. It feels like I can pry it apart, although it's a sealed case and extensive probing doesn't reveal any obvious seams. It just feels … cheap!

It took me a bit to figure out how to turn the unit on. There are two switches on one edge (top? Bottom? Hard to say), both identical. One turns the unit on and off. the other one is the “Home” button.

Using it, It seems slugish as I'm using it. I've never been fond of Android (having been spoiled by the iPhone) but this thing is just painful to use. The interface is confusing (and again, I'm not sure how much of this is Android and how much of this is the Mi Traveler changing things up to “add value”) and the tentacles of Google are very much evident here—no I do not want to use the Google ~~Butt~~ Cloud thank you very much. Then there's the YouTube app. Why does it need my contacts, location, use of the camera and microphone and the ability to send an SMS? Bad Google! No cookie for you!

And now we get to the camera.

It's bad. It's really bad.

My iPhone takes better pictures. So does the iPad.

In fact, to get a picture as bad as the Mi Traveler, I had to crank the quality of the image down to 10%, and even then, I still think the picture is better than the one from the Mi Traveler.

It might be fun to hack on though.

Now … where did I put my pick-axe?

Friday, August 24, 2018

Musings on the intersectionality of trigger warnings and spoiler warnings

I was reading XXX XXXXXXXXX XXXXX XXXX XX XXXXXXX [Hold it right there! You need a trigger warning for that link! –Editor] [Sigh. Fine. –Sean]

(TRIGGER WARNING: XXXXXXXXXX XXXXXXX XXXXXXX) [STOP! SPOILER! Where's the spoiler warning? –Editor] {Seriously? I was trying to give a trigger warning, and now you want a spoiler warning as well? –Sean] [Yes. We can't just go XXXXXXXX X XXXXX XXXXXX! –Editor] [Isn't that a spoiler as well? You can't say that if I can't say XXXXXXXXXX XXXXXXX XXXXXXX –Sean] [… –Editor] [Well? –Sean] [Deep subject. But yes, you are correct. Carry on. –Editor]

SPOILER WARNING: The following trigger warning contains a spoiler for …

[Okay, so how I am supposed to do this? I need to both give a spoiler warning and a trigger warning for the page, but the trigger warning contains a spoiler. If you don't want to get spoiled, you could potentially get triggered. But if I give the trigger warning, those who hate spoilers will have the story ruined! What now, high and mighty editor? –Sean]

[Well, you are well and truly XXXXXX! –Editor]

[Trigger warning for the foul language, you XXXXXXX! –Sean]

Some musings on a fable of software engineers

[You might want to read this first. –Editor] [Really? –Sean] [Really. Or don't. But don't say I didn't warn you. –Editor] [Sigh. –Sean]

I was reading The Codeless Code: Case 41 Garbage (link via Lobsters) and my initial thought was that the head monk might not have much experience with source control. I know I only first started using source control back in 2000 (CVS) and not to control multiple versions but to allow an easy method to update the code from a remote location. Before the use of source control, I found myself commenting out code to keep it “just in case.”

These days, I no longer keep commented out code (erm … um … okay, I should probably just remove the code and convert the information into a real comment) but the thought did strike me that, okay, I remove the code but I have to remember the code existed at one point. Or a maintainer will have to be cognizant enough to search the history looking for deleted code that could be useful. There's very little indication of code that's been removed.

Just a thought.