A con man has been carefully planning his scam for months. After getting a credit card in a phony name, he makes a series of purchases, followed by payments, and he then watches and waits as his spending limit rises. Finally the day comes for him to “bust out,” an industry term referring to a customer charging a card to the limit with resalable items and vanishing. But when the thief goes to the checkout, his card is declined. How was he found out?
The answer has something to do with Netflix. Yes, Netflix. This real-life situation reveals just one of the areas in which technology perfected to win the Netflix Prize—a $1 million contest designed to improve the online movie-rental site’s recommendation system—is extending its influence. We generate reams of data every day, from supermarket trips and Internet browsing to Facebook posts and doctor visits. In ways no one predicted, the prize has accelerated the growth of the industry that tries to analyze and monetize our rapidly multiplying digital traces.
Netflix was cofounded in 1997 by mathematician and software engineer Reed Hastings and direct-marketing guru Marc Randolph; one year later, they began mailing DVDs to subscribers. Pretty quickly, the men realized that if the website could be programmed to act similar to a savvy video-store clerk—directing people to movies they loved, not just ones they liked—subscribers would keep returning. But unlike a Blockbuster employee who had an entire store’s worth of VHS movies to refer to, Netflix’s business rested entirely on DVDs, and the number of titles on the new format was limited. So the company’s virtual clerk had to really know customers’ tastes to match them to the thin catalog of films.
Hastings and Netflix’s recommendations chief, Stan Lanning, developed a program called Cinematch. It relied on a “nearest neighbor”–type algorithm, or a mathematical formula, that clustered customers with similar tastes and examined all their ratings to create matrices of “neighbor” films. For example, a subscriber who had rated Apocalypse Now as a five-star film would be steered to movies he hadn’t seen yet but that had scored well with other members in the five-star Apocalypse Now group.
Cinematch was a hit. It became good enough at forecasting customers’ likes that eventually most movies rented came from its prompts. But by 2006, it had hit a virtual wall in terms of its accuracy in predicting subscriber preferences, despite tinkering by Hastings and his engineers. Inspired by the British government’s Longitude Prize in the 18th century that perfected the chronometer and revolutionized navigation, Hastings decided to hold his own 21st-century public challenge. The winning team or person would receive $1 million for being the first to create an algorithm that improved on Cinematch by 10 percent or more; additional $50,000 Progress Prizes would be awarded to the leaders at one-year intervals. The prize was open to anyone from any country allowed to do business with the U.S.
Besides the cash, Netflix had a potent bit of computer-geek catnip to dangle: data. Since the contestants would need statistics to write their algorithms, the company was releasing 100 million ratings made on 18,000 movies by 480,189 members. For many programmers and mathematicians, the chance to work with the largest trove of real consumer data ever published was all the inducement they needed to jump in.
The Netflix Prize kicked off in the fall of 2006, and by the contest’s end in the summer of 2009, it had attracted more than 51,000 contenders from 186 countries. It fostered a high degree of fierce yet friendly competition. Much of that was due to the fact that it was held almost entirely in the open, with teams tracked on a Netflix leader board, where they’d post their latest solutions. The site also hosted forums in which participants bonded (like about the one subscriber in the data et who’d rated more than 17,000 movies) and shared their latest equations, methods and theories. And as the field dwindled, the boards became a place for rivals to meet and form alliances in an attempt to win.
The first Progress Prize was won in October 2007 by a team called BellKor, three data-mining experts from AT&T Labs. They put in about 2,000 hours to craft a formula that combined 107 algorithms to improve on Cinematch by 8.43 percent. Over the next two years, however, progress slowed to fractions of a percent, as scientists reanalyzed the data to identify the infinite variables that affected people’s ratings. Netflix officials wondered if the prize could ever be won.
Then, in July 2009, BellKor’s Pragmatic Chaos—which consisted of the men of team BellKor, a pair of software developers from Montreal and two Austrian computer-science researchers—came up with a formula that surpassed the 10 percent mark set by Hastings. Much to their shock, so did the Ensemble, an international group of 35 scientists. Netflix officials said the deciding factor would be time, and after a 34-monthlong contest, the Ensemble lost out on $1 million because the group turned in its solution 20 minutes later than the winning team.
Netflix ended up using two of the algorithms produced to refine Cinematch, and they’re still being used today. But surprisingly, the company never adopted the million-dollar formula. That’s because by the time the contest ended, its business model had shifted away from DVD rentals and to streaming video. Now programmers had so much more information than ratings and dates to assess: How much of a video did a customer watch before stopping? When was it stopped? How many videos did a person watch in a sitting? Did viewing preferences change when a subscriber watched on a phone vs. computer vs. Xbox? Still, the advances in algorithms driven by the Netflix Prize are proving to be invaluable in grappling with this influx of data.
Strangely enough, the contest’s greatest effect was not on Cinematch but on the rest of the business world, and that impact is still playing out. More people swarmed into the field of data analytics, because as AT&T’s Robert Bell, a member of the winning team, put it, “winning the prize showed you don’t need a lot of heavy training and background.” More important, “the contest emphasized the power of algorithms in making accurate predictions,” said Jacob Spoelstra, head of R&D for Opera Solutions, a New York–based predictive-analytics firm and one of the main sub-teams in the Ensemble. “Often big organizations struggle with making sense of all the data they have.”
At AT&T, the winners employed their victorious algorithm to detect customer fraud and to suggest TV shows for subscribers to its U-verse digital cable and Internet service. Opera Solutions used its secondplace finish and the knowledge gained in the process to transform itself into one of the top U.S. businesses churning through Big Data. By the prize’s conclusion, CEO Arnab Gupta estimated he’d already seen a $10 million internal payoff.
Since then, Opera Solutions has been hired by a wide variety of private- and public-sector clients. It helped a hospital client crunch years of ER data to discover that patients were more likely to go in on weekends from 7 a.m. to 10 a.m. and from February to April. Thanks to the analysis, the hospital redeployed its nurses and cut labor costs by 18 percent. For a retailer, the company devised an algorithm that scoured millions of credit card receipts to sniff out thieves. It learned that cardholders who suddenly change where they shop or what they buy, or who make expensive purchases of resalable items such as electronics, or who shop at times unusual for them might be on the verge of a “bust out.”
Joe Sill, a member of the Ensemble, launched a consulting business out of his Netflix Prize fame and is now applying science to make predictions in the prosports arena; he’s working with the NBA’s Washington Wizards. Sill believes that scouring e-mails and social networks Facebook, Twitter and Tumblr for nontraditional data, like comments, postings and looking at whom particular people interact with and how, is the next frontier for predictive algorithms.
In August 2009, basking in the glow of the contest’s success, Netflix announced a second prize. This time, it said participants would have an even richer data set—they’d also receive subscriber ages, genders, zip codes, genre ratings and previously chosen movies.
But two months later, a class-action lawsuit, Doe v. Netflix, was filed in federal court, and it tossed out a bombshell of a claim: that the first Netflix Prize “perpetrated the largest voluntary privacy breach to date, disclosing sensitive and personal identifying consumer information.” Plaintiff Jane Doe of Colorado was a closeted “lesbian who does not want her sexuality nor interests in gay and lesbian themes broadcast to the world,” and her lawyers alleged that her sexual preferences and identity could have been gleaned from the company data released—a subscriber ID number, movie title, the subscriber’s star rating for a film and the date of the rating.
And her lawyers had proof: Two University of Texas researchers wrote an algorithm that compared all of the anonymous Netflix subscribers and their ratings with 50 Imdb.com users, and with that small pool, they said they could identify two individuals. On the heels of the lawsuit, the Federal Trade Commission (FTC) sent a letter probing Netflix’s protection of its members’ privacy.
By March 2010, Netflix had settled the suit, reached an understanding with the FTC and scuttled a second prize. What continues to concern privacy advocates about the Netflix-style release of our online information is that we view much of it, such as movie ratings and Facebook likes, as innocuous. Instead, we focus our worries on our Social Security, credit card or passport numbers getting out. But Jane Doe’s example shows that seemingly insignificant details can end up revealing secrets. Also troubling is that we may only know when or if such an ID’ing happens after the truth has already come out. “It’s like a trial where you are being processed, charged and evaluated in ways you don’t know about and don’t have the opportunity to respond to,” says Lee Tien of the Electronic Frontier Foundation.
The Netflix Prize has galvanized other organizations to hold public science contests, but the officials involved with them have said they’re 
taking every precaution to prevent the unmasking of individuals. The Heritage Provider Network, a California physicians group, kicked off a two-year, $3 million competition in the spring of 2011, and it hired as a privacy consultant one of the Texas researchers who’d de-anonymized the Netflix data. The winner of the Heritage Health Prize will be the one who creates an algorithm that successfully predicts how many days a patient will spend in a hospital in the next year. Such a formula could have huge implications, both for our health care system and our economy—it could save some of the billions of dollars estimated to be a year spent on unnecessary hospitalizations in our country. Opera Solutions is currently near the top of the leader board.
Gina Keating’s book Netflixed: The Epic Battle for America’s Eyeballs (Portfolio Books) has just been released.
