The Boston Diaries

Friday, August 10, 2018

Just a small observation about an email I received at work

I'm checking the work email and lo, there's email from our Vice President of Product Management requesting us, unironically, to wear our Corporate flair.

Tuesday, August 07, 2018

Notes on an overheard conversation in a typical company lunch room

“Nice music.”

“It's classical.”

“Classical? That's modern jazz.”

“It's Gershwin. He's white, and he's dead. Therefore, it's classical.”

“So is Kurt Cobain, but I don't consider his music classical.”

Oh wait … I see now … send a completely crazy email so I'll post it

I get quite a bit of spam asking to add links to old entries. It's clear that they do a search for something like “Lost Wages” and spam the XXXX out of people who link to any page dealing with their search term. But the one I received yesterday …

From

Adam Conrad <adam@shihtzuexpert.com>

To

sean@conman.org

Subject

Article Contribution on Boston.Conman.Org

Date

Mon, 6 Aug 2018 23:38:14 -0700

Sean,

I really like the post on your site at http://boston.conman.org/2007/12/27.1

I’ve also, been writing about Shih Tzu and other small, toy or medium breed dogs and related issues like health, food etc at

The Shih Tzu Expert

http://shihtzuexpert.com/canine-distemper/

I would love the chance to write a unique and an interesting guest post for your blog.

I can write about any topic related to pets.

Our articles are 700+ words, written by native English speakers/writers.

Our content is unique, fresh, detailed, and thoroughly researched.

If it covers medical issues or advice, we always require our in house veterinarian to check accuracy of facts before publication.

A selection of titles we are proposing

What to look for when buying dog food for Shih Tzu and other toy breeds

Grooming advice, How to groom, best hair cuts, best grooming products.

Veterinary approved articles about Health issues, Distemper, Parvo, Obesity, Lack of appetite etc

If there is any topic you want us to cover, please feel free to let us know.

Please feel free to contact me if you have any questions.

Best regards,

Adam Conrad

ShihTzuExpert.com

Seriously … WTF?

The page Adam linked to?

Nothing to do with dogs!

I have entries where I talk about dogs. And cats. And dogs and cats. It's not like Adam lacked entries to choose from.

But nooo! Adam choose one sans dogs.

What?

All I'm asking for is some consistency between APIs and version numbers

When I first started working with libtls, I thought that TLS_API designated a change in API so that one could work with different versions of the library without breaking the compilation process. Sadly, that wasn't the case, so I switched to using LIBRESSL_VERSION_NUMBER, as that seemed to regularly change with each version.

I was doing this so that my Lua wrapper could be compiled with any version of libtls. Why break things unnecessarily? And things were going fine until I hit version 2.2.1, and well …

Mismatches in `libtls` between functions added, `LIBRESSL_VERSION_NUMBER` and `TLS_API`
Version	Functions	`LIBRESSL_VERSION_NUMBER`	`TLS_API`
Version	Functions	`LIBRESSL_VERSION_NUMBER`	`TLS_API`
2.1.2	`tls_accept_socket()` `tls_client()` `tls_close()` `tls_config_clear_keys()` `tls_config_insecure_noverifycert()` `tls_config_insecure_noverifyhost()` `tls_config_set_ca_file()` `tls_config_set_ca_path()` `tls_config_set_cert_file()` `tls_config_set_cert_mem()` `tls_config_set_ciphers()` `tls_config_set_ecdhcurve()` `tls_config_set_key_file()` `tls_config_set_key_mem()` `tls_config_set_protocols()` `tls_config_set_verify_depth()` `tls_config_verify()` `tls_configure()` `tls_connect()` `tls_connect_fds()` `tls_connect_socket()` `tls_error()` `tls_free()` `tls_init()` `tls_read()` `tls_reset()` `tls_server()` `tls_write()`	0x20000000	20141031
2.1.4	`tls_load_file()`	0x20000000	20141031
2.2.0	`tls_accept_fds()`	0x20000000	20141031
2.3.0	`tls_config_insecure_noverifytime()` `tls_config_prefer_ciphers_client()` `tls_config_prefer_ciphers_server()` `tls_config_verify_client()` `tls_config_verify_client_optional()` `tls_conn_cipher()` `tls_conn_version()` `tls_handshake()` `tls_peer_cert_contains_name()` `tls_peer_cert_hash()` `tls_peer_cert_issuer()` `tls_peer_cert_provided()` `tls_peer_cert_subject()` `tls_read()` (paramter change) `tls_write()` (parameter change)	0x20030000	20141031
2.3.1	`tls_peer_cert_notafter()` `tls_peer_cert_notbefore()`	0x20030001	20141031
2.4.0	`tls_config_keypair_file()` `tls_config_keypair_mem()`	0x2040000f	20141031
2.5.0	`tls_accept_cbs()` `tls_config_add_keypair_file()` `tls_config_add_keypair_mem()` `tls_config_alpn()` `tls_conn_alpn_selected()` `tls_conn_servername()` `tls_connect_cbs()`	0x2050000f	20160904
2.5.1	`tls_ocsp_process_response()` `tls_peer_ocsp_cert_status()` `tls_peer_ocsp_this_update()` `tls_peer_ocsp_url()` `tls_config_add_keypair_ocsp_file()` `tls_config_add_keypair_ocsp_mem()` `tls_config_add_ticket_key()` `tls_config_keypair_ocsp_file()` `tls_config_keypair_ocsp_mem()` `tls_config_ocsp_require_stapling()` `tls_config_ocsp_staple_file()` `tls_config_ocsp_staple_mem()` `tls_config_session_id()` `tls_config_session_lifetime()` `tls_peer_ocsp_crl_reason()` `tls_peer_ocsp_next_udpate()` `tls_peer_ocsp_response_status()` `tls_peer_ocsp_revocation_time()`	0x2050100f	20170126
2.6.0	`tls_config_crl_file()` `tls_config_crl_mem()` `tls_peer_cert_chain_pem()` `tls_unload_file()`	0x2060000f	20170126
2.6.1	`tls_config_echdecurves()`	0x2060100f	20170126
2.7.0	`tls_config_session_fd()` `tls_conn_session_resumed()`	0x2070000f	20180210

I'm not asking for much. I'm not asking for slavish adherance to semantic versioning. I'm just asking for a consistent way to check an API to I can support earlier versions of a library.

Don't get me wrong, I'm glad that libtls exists, and as an API, it's much nicer than the eldritch horror of OpenSSL.

I just wish they had updated TLS_API (or LIBRESSL_VERSION_NUMBER) consistently. Otherwise, why have them in the first place?

Monday, August 06, 2018

It seems that checking the TLS API version number is useless

I've pretty much finished the Lua TLS module and before releasing it, I thought it might be nice to ensure it compiles with previous versions of libtls. The main header file contains the defined value TLS_API, which I assume is updated whenever the API is updated. So I began the arduous procedure of downloading previous versions of libtls to ensure I can compile against any version.

I started with LibreSSL version 2.7.4 (current when I started—they are now up to 2.8.0 as I write this). The defined value TLS_API had a value of “20180210”. I checked version 2.7.0 and no change in libtls. It wasn't until I got into the pre-2.7 versions that things started going south.

The previous version of TLS_API, “20170126”, was first defined in 2.5.1, and last used in 2.6.5. But the API changed quite a bit between versions 2.5.1 and 2.6.5. Five functions were added:

tls_config_set_crl_file()
tls_config_set_crl_mem()
tls_config_set_ecdhecurves()
tls_peer_cert_chain_pem()
tls_unload_file()

What's the point of having a defined value like TLS_API if it doesn't change when you add new functions?

Fortunately, the defined value LIBRESSL_VERSION_NUMBER is updated per version, so at least I can use that.

Sigh.

I guess we'll see how well CBOR works

The Corporate Overlords of The Corporation have spoken, and lo, they said “Worry not, for you do not have to deal with the stupidity—we shall deal.” So great, we don't have to drink the REST/HTTPS über alles Kool-Aid™, but instead query our Corporate Overlords for the data, who have drunk the REST/HTTPS über alles Kool-Aid™ (or were forced to by the company we query for the data—the end effect is the same though).

Sigh.

So for now, we still make our queries via UDP, only now in CBOR—the legacy format of DNS is apparently too arcane to support any more.

I have a Lua module but we also need one in C. There are some existing ones, but they have their issues (either an alien build system or missing some critical CBOR feature) so I've been working on a C library I started a few years ago and never finished.

It's working now, and to test it, I've been using valgrind to ensure memory safety, in addition to using /dev/urandom to generate random garbage:

GenericUnixPrompt% dd if=/dev/urandom count=2 | tee bad-data | valgrind ./testdecoder

dd is one of those relatively arcane Unix programs that I find useful on occasion (like here to generate some random data). tee I use to record the data so I can play it back when valgrind finds an issue. This is a reasonable way to fuzz a program. It did find several issues that could have lead to a crash, and I don't leak any memory so the code should be good to go.

Monday, July 23, 2018

Managing TLS connections using Lua and Lua coroutines

Getting libtls working with Lua wasn't as straightforward and I thought it would be. It works (for the most part) but I had to change my entire approach. The code is an ugly mess and there's quite a bit of duplication in several spots.

But! I can request web pages, in Lua, via HTTPS in an event loop based around select() (or poll() or epoll() or whatever is the low level event notification scheme used). Woot! And I'm going into excruciating detail on this.

Back on Friday, when I wrote some “proof-of-concept” code, I had thought I could switch coroutines in the user-supplied I/O callback routines (and if coroutines existed in C, that is where you would yield to another coroutine). It was easy enough to extend the callback to a Lua routine— in the routine that wraps the libtls function tls_connect_cbs():

static int Ltls_connect_cbs(lua_State *L)
{
  struct tls **tls = lua_touserdata(L,1);
  int rc           = tls_connect_cbs(
			*tls,
			Xtls_read,
			Xtls_write,
			L,
			luaL_checkstring(L,2)
		     );
  
  if (rc != 0)
  {
    lua_pushboolean(L,false);
    return 1;
  }
  
  lua_settop(L,5);
  lua_pushlightuserdata(L,*tls);
  lua_getuservalue(L,1);
  lua_pushvalue(L,1);
  lua_setfield(L,-2,"_ctx");
  lua_pushvalue(L,2);
  lua_setfield(L,-2,"_servername");
  lua_pushvalue(L,3);
  lua_setfield(L,-2,"_userdata");
  lua_pushvalue(L,4);
  lua_setfield(L,-2,"_readf");
  lua_pushvalue(L,5);
  lua_setfield(L,-2,"_writef");
  
  lua_settable(L,LUA_REGISTRYINDEX);
  lua_pushboolean(L,true);
  return 1;
}

I pass in the two callback functions, and I'm using the Lua state context as the userdata in the callbacks. I then create a Lua table, populate it with some useful information, such as the Lua functions to call, and associate it in the Lua registry with the value of the libtls context. Then, when libtls calls one of the callbacks:

static ssize_t Xtls_write(struct tls *tls,void const *buf,size_t buflen,void *cb_arg)
{
  lua_State *L = cb_arg;
  ssize_t    len;
  
  lua_pushlightuserdata(L,tls);
  lua_gettable(L,LUA_REGISTRYINDEX);
  lua_getfield(L,-1,"_writef");
  lua_getfield(L,-2,"_ctx");
  lua_pushlstring(L,buf,buflen);
  lua_getfield(L,-4,"_userdata");
  lua_call(L,3,1);
  
  len = lua_tonumber(L,-1);
  lua_pop(L,2);
  return len;
}

I get the Lua state via the user argument. From that, and the libtls context, I obtain the data I cached into the Lua table, which give me the Lua function to call. Said function can then call coroutine.yield().

Straightforward, easy, and wrong! I got the dreaded “attempt to yield across metamethod/C-call boundary” error. Darn.

The attempted flow looks like (yellow boxes are Lua functions; green boxes are C functions):

${data=tls.read()} → [Ltls_read(lua)] → [tls_read(ctx)] → [Xtls_read(ctx,lua)] → [lua_call(lua)] → {my_callback()} → {coroutine.yield()} {}=Lua function []=C function$

There are four layers of C functions that can't be yielded through. Lua does have a way of dealing with intervening C functions, but it's somewhat clunky.

${luaf_a()} → [cf_orig(lua)] → [lua_callk(lua,cf_c)] → {luaf_b()} → {coroutine.yield} / {coroutine.resume} → {luaf_b()*} → [cf_c(lua)*] → {luaf_a()}$

In this case, the Lua function lua_callk() is handled specially so it doesn't cause an error. The function cf() needs to be split in half—the portion prior to calling into Lua, and the second half to handle things after a potential call to coroutine.yield(). That's represented above by the functions cf_orig() and cf_c(). The “*” represent the functions returning, not calling. coroutine.resume() will restart luaf_b() right after it's call to coroutine.yield(). And when luaf_b() returns, it “returns” to cf_c(), which does whatever and finally returns, which “returns” to luaf_a().

But in the case I'm dealing with just doesn't work with that model. The code calling into Lua doesn't have the signature:

int function(lua_State *lua_State);

but the signature:

ssize_t function(struct tls *ctx,void *buf,size_t buflen,void *cb_arg);

Not only are the return types different, but they have completely different semantics—for libtls, it's the number of bytes transferred, whereas for Lua, it's the number of items being returned to Lua.

No, I had to rethink the entire approach, and do the call to coroutine.yield() a bit higher in the call stack. Which also meant I had to push dealing with TLS_WANT_POLLIN and TLS_WANT_POLLOUT back to the caller. The documentation states:

TLS_WANT_POLLIN The underlying read file descriptor needs to be readable in order to continue.

TLS_WANT_POLLOUT The underlying write file descriptor needs to be writeable in order to continue.

In the case of blocking file descriptors, the same function call should be repeated immediately. In the case of non-blocking file descriptors, the same function call should be repeated when the required condition has been met.

And here I was, trying to hide such concerns from the user. Ah well.

I eventually got it working, but man, is it ugly. The Lua code wants to read data, so I have to call into libtls. That in turn, calls back into my code, and if I don't have any, I need to return TLS_WANT_POLLIN, which bubbles up through libtls back to my code, which can then yield.

Meanwhile, from the other end, I get data from the network. I can't just feed it into libtls, I have to feed it when libtls calls the callback for the data. But when I get the data, I may need to resume the coroutine, so I have to track that information as well.

I can almost understand the code (and yes, I wrote it; did I mention it's ugly?)

But I'm happy. The following code works in my existing network framework (boy does that sound wierd):

local function request(item)
  syslog('debug',"requesting %s",item.url)
  local u = url:match(item.url)

  -- -------------------------------------------------------
  -- asynchronous DNS lookup---blocks the current coroutine
  -- until a result is returned via the network.
  -- -------------------------------------------------------

  local addr = dns.address(u.host,'ip','tcp',u.port)
  
  if not addr then
    syslog('error',"finished %s---could not look up address",u.host)
    return
  end

  -- ---------------------------------------------------------
  -- This has nothing to do with the iPhone operating system,
  -- but everything to do with "Input/Output Stream"
  -- ---------------------------------------------------------

  local ios
  
  if u.scheme == 'http' then
    ios = tcp.connecta(addr[1]) -- connect via TCP
  else
    ios = tls.connecta(addr[1],u.host) -- connect via TLS
  end
  
  if not ios then
    syslog('error',"could not connect to %s",u.host)
    return
  end
  
  local path    = table.concat(u.path,'/')
  local fhname  = "header/" .. item.hdr
  local fbname  = "body/"   .. item.body
  local fh      = io.open(fhname,"w")
  local fb      = io.open(fbname,"w")
  
  local command = string.format([[
GET /%s HTTP/1.0
Host: %s
Connection: close
User-Agent: TLSTest/2.0 (Lua TLS Testing Program)
Accept: */*

]],path,u.host)

  ios:write(command)
  
  fh:write(ios:read("*h"))
  
  repeat
    local data = ios:read("*a")
    fb:write(data)
  until data == ""
  
  fb:close()
  fh:close()
  ios:close()

  syslog('debug',"finished %s %s",item.url,tostring(addr[1]))
end

Any number of requests can be started and they all run concurrently, which is just what I wanted.

Now, the code I have for the Lua wrapper for libtls covers just what I need to do this. More work is required to finish covering the rest of the API. I also have to clean up the Lua code that backs the above sample code so that I might have a chance of understanding it at some point in the future.

And until I get the working code published, you can look at the “proof-of-concept” Lua coroutine code I worked from (and no, the above code sample will not work as is with this “proof-of-concept” code).

Thursday, July 19, 2018

A sane and easy to use TLS library! Will wonders never cease!

I'm still fighting the stupidity at work, but it's becoming aparent that it's a fait accompli and we're looking at a bunch of REST/HTTPS über alles Kool-Aid™ in an area where time is critical.

Sigh.

So I'm looking around at what I can use to support the “S” in HTTPS that doesn't involve diving into the horror show that is OpenSSL. A library that can still encrypt and decrypt data when it isn't managing the network connections on the program's behalf (because the program is already managing the network connections). It can be complicated, but it must be sane to use.

I was pointed to libtls, which comes with LibreSSL. Not only is this sane, but it's easy to use. I'm simply amazed at how easy.

In just an hour, and only reading the man pages, I was able to write a simple program that fetches a page from a secure website. And most of the code is just there to report any errors that happen. It's a very straight forward program.

Another hour or two, and I had a program where the library does not control the network connection. Which means we can (probably) use this in our existing architecture.

A few more hours, and I was able to replicate the initial C program in Lua:

local tls = require "org.flummux.tls"

-- *****************************************************************

local function okay(v,err)
  if not v then
    print(">>>",err)
    os.exit(1)
  end
  return v
end

-- *****************************************************************

if #arg == 0 then
  io.stderr:write(string.format("usage: %s host resource\n",arg[0]))
  os.exit(1)
end

local config = tls.config()
local ctx    = tls.client()

okay(config:set_protocols "all")
okay(ctx:configure(config))
okay(ctx:connect(arg[1],"https"))
okay(ctx:write(string.format(
     "GET %s HTTP/1.1\r\n"
  .. "Host: %s\r\n"
  .. "User-Agent: TLSTester/1.0 (TLS Testing Program Lua)\r\n"
  .. "Connection: close\r\n"
  .. "Accept: */*\r\n"
  .. "\r\n",
     arg[2],
     arg[1]
)))

while true do
  local bytes = okay(ctx:read(1024))
  if bytes == "" then break end
  io.stdout:write(bytes)
end

I had to write my own Lua wrapper for LibreSSL. The existing ones (and I found only two) weren't up to my standards for use, but it wasn't terribly hard to get the above working.

The next step is expanding the Lua module to see if I can get it working with our networking code we use. I am optimistic about this.

But I am not optimistic about having to use this at work.

Monday, July 16, 2018

I don't think this is what McDonald's had in mind when they said “you deserve a break today”

I was at The Scottish Place to pick up some lunch. I placed my order, and while waiting in line, I noticed the telltale signs of Microsoft Windows:

[Image of a crashed Windows box: A stopped clock is right twice a day, unless it's one of those 24-hour clocks which makes it right only once a day, but a crashed Windows box isn't right at any time of day (and some would say that even a functioning Windows box isn't right at any time of the day, but I digress ...)]

Ah, Windows, keep on being your special snowflake self.

But in all seriousness, who set this up? If you are going to run any computer in a public space, I would hope those setting it up would run the bare minimum for the service required (and in this case, just the operating system and whatever program it is that runs the display) and remove everything else. Then, for whatever left is running, is there not some Windows setting to just have it automatically reboot upon an error? Or simply not display warning messages like this on the display meant for public viewing?

I have to wonder how long this message has been here, waiting for someone, anyone, to select an option (or to restart, reboot or reinstall). I wonder if management even cares?

Monday, July 09, 2018

Yes. It is music. I identified it. What more do you want?

Bunny and I had dinner at a nearby Chinese restaurant. As we ate, Bunny mentioned that the music we were listening to must be the Chinese equivalent to elevator music. I concurred, and because I was curious to know what it was, I thought I might try to use Shazam to identify it.

It did.

And now I know.

At least now no one has to pay to have their information protected

So I received the following email from my registrar:

From

<XXXXXXXXXXXXXXXXXXXXXXX>

To

<sean@conman.org>

Subject

WHOIS Data Confirmation for x-grey.com

Date

Mon, 9 Jul 2018 09:30:23 -0400 (EDT)

Dear Valued Customer,

ICANN, the organization responsible for the stability of the Internet, requires that each domain name registrant be given the opportunity to correct any inaccurate contact data (WHOIS data) associated with a domain name registration. Our records for your domain are as follows:
x-grey.com

Domain Name: X-GREY.COM
Registry Domain ID: 1325416434_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.domain.com
Registrar URL: www.domain.com
Updated Date: 2017-11-08T05:11:47
Creation Date: 2017-11-07T22:32:17
Registrar Registration Expiration Date: 2018-11-08T01:32:21
Reseller: Dotster.com
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
Registry Registrant ID:
Registrant Name: Data Protected Data Protected
Registrant Organization: Data Protected
Registrant Street: 123 Data Protected
Registrant City: Toronto
Registrant State/Province: ON
Registrant Postal Code: M6K 3M1
Registrant Country: CA
Registrant Phone: +1.0000000000
Registrant Phone Ext:
Registrant Fax: +1.0000000000
Registrant Fax Ext:
Registrant Email: noreply@data-protected.net
Registry Admin ID:
Admin Name: Data Protected Data Protected
Admin Organization: Data Protected
Admin Street: 123 Data Protected
Admin City: Toronto
Admin State/Province: ON
Admin Postal Code: M6K 3M1
Admin Country: CA
Admin Phone: +1.0000000000
Admin Phone Ext:
Admin Fax: +1.0000000000
Admin Fax Ext:
Admin Email: noreply@data-protected.net
Registry Tech ID:
Tech Name: Data Protected Data Protected
Tech Organization: Data Protected
Tech Street: 123 Data Protected
Tech City: Toronto
Tech State/Province: ON
Tech Postal Code: M6K 3M1
Tech Country: CA
Tech Phone: +1.0000000000
Tech Phone Ext:
Tech Fax: +1.0000000000
Tech Fax Ext:
Tech Email: noreply@data-protected.net
Registry Billing ID:
Billing Name: Data Protected Data Protected
Billing Organization: Data Protected
Billing Street: 123 Data Protected
Billing City: Toronto
Billing State/Province: ON
Billing Postal Code: M6K 3M1
Billing Country: CA
Billing Phone: +1.0000000000
Billing Phone Ext:
Billing Fax: +1.0000000000
Billing Fax Ext:
Billing Email: noreply@data-protected.net
Name Server: ns1.armigeron.com
Name Server: ns2.armigeron.com
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
>>> Last update of WHOIS database: 2017-11-08T05:11:47 <<<
“For more information on Whois status codes, please visit https://icann.org/epp”

Registration Service Provider:
    XXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXX
    +1.XXXXXXXXXX
    This company may be contacted for domain login/passwords,
    DNS/Nameserver changes, and general domain support questions.

To review and update your WHOIS contact information, please log into our management interface at: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

If any of the information above is inaccurate, you should correct it. If all of the information above is accurate, you do not need to take any action.

Please remember that under the terms of your registration agreement, the provision of false WHOIS information can be grounds for cancellation of your domain name registration. If you have any questions or comments regarding ICANN's policy, please visit the following link http://www.icann.org/registrars/wdrp.htm

Thank you for your attention.

Best regards,
XXXXXXX

It's kind of hard to review my information when it's protected from my viewing it for my protection. I think my registrar is going a bit overboard in complying with the GDPR.

Sigh.

Friday, August 10, 2018

Just a small observation about an email I received at work

Tuesday, August 07, 2018

Notes on an overheard conversation in a typical company lunch room

Oh wait … I see now … send a completely crazy email so I'll post it

All I'm asking for is some consistency between APIs and version numbers

Monday, August 06, 2018

It seems that checking the TLS API version number is useless

I guess we'll see how well CBOR works

Monday, July 23, 2018

Managing TLS connections using Lua and Lua coroutines

Thursday, July 19, 2018

A sane and easy to use TLS library! Will wonders never cease!

Monday, July 16, 2018

I don't think this is what McDonald's had in mind when they said “you deserve a break today”

Monday, July 09, 2018

Yes. It is music. I identified it. What more do you want?

At least now no one has to pay to have their information protected

Obligatory Picture

Obligatory Links

Obligatory Miscellaneous