Wednesday, February 20, 2013
“Just sell public information,” they said …
If we're offering advice from the peanut gallery (and I'm no smartphone developer, so I'm way up in the peanut gallery), why not spend that time on a program that can generate trivial apps.
Extract data [legally, now!] from some source (eg wikipedia), and skin it with some boilerplate and some generated UI elements. I think it'd be a more interesting project to implement the more general version, and once it's nailed down it's probably a better source of revenue too.
If we're offering advice from the peanut gallery (and I'm no smartphone develope… | Hacker News
This reminds me of a job I had years ago.
And given that it was probably over ten years since I did this job, I think I'm now safe enough to talk about the job, which involved scraping public information from a website so my employer could sell the information.
It was a freelance job. The gentleman who hired me was a lawyer specializing in the financial industry. The idea was to download public information, package it up as a book to sell it to financial market insiders. All I needed to do was to scrape the public information from a website.
The Public Disclosure Program discloses the following information on firms:
- the firm's name, CRD number, SEC identification number, “applicant name” or legal name, NASD district office which oversees the firm, and the firm's main and mailing addresses
- the types of business in which the firm is currently engaged
- details of the firm's legal status (i.e, corporation, partnership, etc.), state or country and date of formation, and its fiscal year end
- all approved registrations
- disclosure matters involving certain criminal charges and convictions, regulatory actions, civil judicial actions, and certain financial actions (e.g., bankruptcies, unsatisfied judgment/liens.)
Easy stuff. Just submit a form, get the output. Pass it along to the lawyer. And when I found out how much he was going to charge (around $1,200 a copy) I was kicking myself for not thinking of this on my own.
But there were problems. First off, the site in question took a dim view of my scraping (even though I wasn't hitting the site all that hard—maybe a request every few minutes) that they changed how the results were returned. Now I had to set up email accounts to accept the results.
Then I had to learn how to manage emails with attachments.
They then changed the sumbmission form multiple times.
In all of this, I was told that the information is public and that there is no question of legality involved with this. Remember, my employer was a lawyer and well … okay, it's public information about securities firms.
So I kept up with all the changes and kept handing over the files to my employer.
Then I received a call from their lawyers.
Oh.
I immediately told them I was just a hired gun and that the person they really wanted to talk to was my employer. Thankfully, I never did hear back from them, and I never had to appear in court nor did I receive a summons. At least my employer kept those lawyers off my back and bore the brunt of a lawsuit against him for selling “public information” (he ultimately lost).
In hindsite, I was mighty glad I didn't have the idea to do that. Not only would I have had trouble selling such a book, given that I knew absolutely nothing about the industry, nor did I know anyone in the industry, but I was shielded from a lawsuit.
Yes, the idea is nice in theory, but in practice, you had an organization that wasn't thrilled with someone actually trying to use the “public information” and made their intentions known.
This is just something to keep in mind if you ever get a similar idea.
