The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Wednesday, February 20, 2013

“Just sell public information,” they said …

If we're offering advice from the peanut gallery (and I'm no smartphone developer, so I'm way up in the peanut gallery), why not spend that time on a program that can generate trivial apps.

Extract data [legally, now!] from some source (eg wikipedia), and skin it with some boilerplate and some generated UI elements. I think it'd be a more interesting project to implement the more general version, and once it's nailed down it's probably a better source of revenue too.

If we're offering advice from the peanut gallery (and I'm no smartphone develope… | Hacker News

This reminds me of a job I had years ago.

And given that it was probably over ten years since I did this job, I think I'm now safe enough to talk about the job, which involved scraping public information from a website so my employer could sell the information.

It was a freelance job. The gentleman who hired me was a lawyer specializing in the financial industry. The idea was to download public information, package it up as a book to sell it to financial market insiders. All I needed to do was to scrape the public information from a website.

The Public Disclosure Program discloses the following information on firms:

Easy stuff. Just submit a form, get the output. Pass it along to the lawyer. And when I found out how much he was going to charge (around $1,200 a copy) I was kicking myself for not thinking of this on my own.

But there were problems. First off, the site in question took a dim view of my scraping (even though I wasn't hitting the site all that hard—maybe a request every few minutes) that they changed how the results were returned. Now I had to set up email accounts to accept the results.

Then I had to learn how to manage emails with attachments.

They then changed the sumbmission form multiple times.

In all of this, I was told that the information is public and that there is no question of legality involved with this. Remember, my employer was a lawyer and well … okay, it's public information about securities firms.

So I kept up with all the changes and kept handing over the files to my employer.

Then I received a call from their lawyers.


I immediately told them I was just a hired gun and that the person they really wanted to talk to was my employer. Thankfully, I never did hear back from them, and I never had to appear in court nor did I receive a summons. At least my employer kept those lawyers off my back and bore the brunt of a lawsuit against him for selling “public information” (he ultimately lost).

In hindsite, I was mighty glad I didn't have the idea to do that. Not only would I have had trouble selling such a book, given that I knew absolutely nothing about the industry, nor did I know anyone in the industry, but I was shielded from a lawsuit.

Yes, the idea is nice in theory, but in practice, you had an organization that wasn't thrilled with someone actually trying to use the “public information” and made their intentions known.

This is just something to keep in mind if you ever get a similar idea.

Obligatory Picture

[“I am NOT a number, I am … a Q-CODE!”]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site:, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2024 by Sean Conner. All Rights Reserved.