Thursday, October 12, 2006
Well, there goes that idea for a million bucks
Netflix is sponsoring a contest to improve it's recommendation engine and in doing so, it releasing 480,000 (scrubbed) customer records to help contestants create better recommendations.
I heard about this a few days ago and I've been mulling over the possibilities, and while I have little experience in writing such engines, that also means I have less preconceived notions of what is and isn't possible, which means I might have a breakthough method that others working on this problem professionally might not even conceive of.
I mean, it could happen, right?
My idea (and yes, I'm giving it away here for reasons that will become apparent in a bit): Netflix allows one to rank genres (for instance, I've ranked “documentaries” highest, followed by “comedies” and “science fiction/fantasy” (in no particular order). My thought—find other people who have ranked genres the same as I and see what they like. I'm also assuming that genre “influence” might bleed over, such that I would prefer a humorous documentary over a more serious one.
For instance, Alice has ranked “science-fiction” highest, followed by “horror.” Bob has ranked “science-fiction” highest, followed by ”action & adventure,” therefore, Alice would probably enjoy “Alien” over “Aliens” as the former is more horror based than the action based latter. While both are classified as “science-fiction” by Netflix as the “primary genre” each has a different “sub-genre” that are picked up by viewers (most likely subconsciously), so Carol, who has ranked genres similarly to Bob would probably prefer “Aliens” over “Alien.” And Dave, who ranked his genre preferences as “science-fiction” and “drama” respectively would probably like “Alien3.”
Well, that was the idea.
I signed up, and downloaded the two gigabytes of data and was disappointed.
You see, they scrubbed the customer data so much, all you get basically, is that on such-n-such a date, customer number so-n-so rated this-n-that film four stars.
That's it.
Okay, you get a list of the 17,000 films and when each customer rated said film, but that's it. No genre information about either.
Granted, I could go in and find the Netflix classification on each film, then try to suss out the genres each customer probably likes, but frankly, that's a whole lot of work for some unreliable reconstruction of data, so I can't even test my idea with any confidence.
And what's to say that Netflix (or rather, Cinematch™) isn't doing that already?
So then I decided to see if I could find anyone working on this contest and blogging about it. In my limited (two) searches, I've only found Harry Chen to have any comments on the contest worth reading, and he makes a point I didn't consider:
Although I have no evidence to prove my theory, but I believe Netflix's user rating data assumes that user ratings came from the same Netflix account were made by a single person. I believe that's not true for most of the Netflix accounts. Many families only subscribe to a single Netflix account. In my family, my wife and I share a single account. We both actively manage this account and rate movies. It's inappropriate to consider our combined ratings as the ratings of a single person. Just because I like action movies and my wife likes comedies, one can't conclude with full confidence that we as a single Netflix account user like both action movies and comedies.
People Rent Movies for Others to Enjoy
Another problem associated with a single account usage is that a user who rents all the movies may not be the same person who actually watches them. For example, parents may rent movies on the behaves of their children. While they rent movies, but they may not watch all of them. In fact, parents may not even enjoy watching them at all. If a parent rents a half dozen of cartoons and 1-2 action movies a month, should the recommendation engine automatically assume that cartoons are the user's favorite? What if the user continues to rent cartoons but at the same time rate those movies with very low scores (1 out of 5 stars), is cartoon still the user's favorite movie?
Why Netflix movie recommendation doesn't work
That might skew the results a bit.
Also, in discussing this with Wlofie, he felt that the single ranking was too limited. Why did I rate, say, “Plan 9 From Outer Space” ★★★★? It's certainly not a good film (worth, maybe, maybe ★) but it's one of those films that are so bad that's it's actually fun to watch in a MST3K way. So having multiple rankings per film can lead to better recommendations.
But by making the ranking system more complicated, it means less uers will bother to rank a film (and really, Netflix has made it dead simple to rank a film—about as simple as you can get it). Heck, even I didn't bother ranking films when I first started at the site. Making it harder means less data. Maybe better data, but does it help if there's so little? (for reference, Wlofie has yet to rank genres he likes at Netflix, and perhaps that's one of the reasons why it scrubbed—not enough people have ranked genres).
It would have been nice to have additional data from Netflix, but given privacy concerns, I can see why they might have scrubbed as much data as they have. Ah well. Perhaps I'll mull this over a bit more and play with the data a bit.
