(This is the third part of a multi-part series exploring one specific Netflix number. To read the other pieces:
– Introducting “Datecdotes”, when Streaming Companies use Data to Win the PR Wars
– Did More People Watch Crazy Rich Asians or a Netflix Rom Com Last Summer?
– Netflix versus Crazy Rich Asians: What Else Does Netflix “80 Million Customer Accounts Tell Us?
– Suspiciously Recurring Numbers and More Implications of The Netflix versus Crazy Rich Asians Debate
– Don’t Cross the Streams: Streaming Video Metrics…Explained!!!)
The romantic comedy is dead. The highest grossing romantic comedy this year is Crazy Rich Asians, which at $174 million in domestic box office pales in comparison to behemoths like Black Panther ($700 million domestic box office), Avengers: Infinity Wars ($678 million), Incredibles 2 ($608 million) and Jurassica World: Fallen Kingdom ($416 million). Who shall return the romantic comedy to glory?
Netflix. Because of course Netflix.
This summer, Netflix released a series of romantic comedies it dubbed, “the summer of love”. In a letter to shareholders, Netflix celebrated their success. Here’s Vox writing about a particular fact Netflix provided:
This summer, Netflix invested in resurrecting the mid-budget romantic comedy, acquiring movies like Set It Up and To All the Boys I’ve Loved Before for what the streaming service branded as its “Summer of Love.” And now, it’s looking like the gamble paid off: Variety reports that more than 80 million subscribers watched one of the 11 rom-coms on the Summer of Love slate, according to Netflix’s quarterly earnings report.
Vox wasn’t alone in singing Netflix’s praises. They were joined by Variety, Screen Rant, The Ringer and others to write an article on Netflix’ new found success in romantic comedy. All using one “datecdote”, a term I coined yesterday.
In our hurry to constantly keep up with the news, we let little tidbits like Netflix’s above fact wash over us and move on to the next story. So let’s pause and reflect on the fact Netflix revealed. Does this fact seem true? Since Netflix didn’t provide a comparison, I will:
Did more people around the globe watch Crazy Rich Asians or a Netflix Romantic Comedy last summer?
It’s a tough question, isn’t it? If you answer that more people saw a Netflix romantic comedy, then why did the media spend so much time on the phenomena of Crazy Rich Asians? But if you think more people saw Crazy Rich Asians, then how can Netflix numbers possibly be true?
Streaming video companies, like Netflix, have a lot of data, a lot of ways they can manipulate that data, and, most crucially, a lot of data they just don’t give the press. But we can learn a lot about how movies are distributed and judged in today’s media landscape by trying to answer that tough question with the data and facts we do have.
So let’s do our best to get some answers.
How Many People Watched a Netflix Romantic Comedy?
On the surface, this is fairly straight forward. Netflix in their letter to shareholders—a document submitted to the SEC, so a legally-binding, carefully vetted document—used this phrase:
More than 80 million accounts have watched one or more of the Summer of Love films globally.
Netflix chose two words very carefully in the above sentence. First is “accounts”. Not “profiles” or “viewers” but accounts, since this is the only unit of measurement Netflix knows for sure. They know that because they have one account per credit card. I’ve seen this called “users” or “customers” at other companies.
By definition, this is the floor for the actual number of people who watched a romantic comedy on Netflix. If two people watched a film together, well they still only count as one “account”. If two different profiles under the same account watched, they would still probably count as one. (It’s unclear.) If someone shares their password with someone else, but they use the same profile, that still counts as one view.
If account is a precise definition, “watched” is a term so loose that it could mean anything. For instance, Netflix could count as “watched”, a person who only watches ten minutes of a film and turns it off. They could only count as “watched” people who watch greater than 80% of a film, either by run time or who watched past the 80% point in the film. We just don’t know.
What we don’t know dwarfs what we little we do know. We don’t know how many total hours of romantic comedies were viewed. (Netflix, interestingly, loves to cite this number to describe how popular their platform is, but choose not to provide that fact here. For example, they released earlier this year that they had 350 million hours viewed in one day in January.) We don’t know where people watched—this is a “global” number—or even when. While presumably over the summer, it likely wasn’t a hard three month window.
How Many People Watched Crazy Rich Asians?
Tracking theatrical box office is data opposite of Netflix streaming data. For box office, we have a few numbers that are very reliable and we have them going back decades. Netflix, on the other hand, has tons of data but they don’t share any of it and it only goes back ten years or so. You probably heard the top line number for Crazy Rich Asians: it generated $174 million in the United States domestic box office and an additional $64 million overseas. (Box office as of 3-December-2018, and rounded up in some cases.)
Of course, for our purposes, the more important number is trying to figure out how many “people” went to go see it. Fortunately, this calculation is fairly easy. Simply take the average ticket price in America and divide by the box office. (In 2018, the average price is $9.14, according to Box Office Mojo) This gets us to this number:
Since its release, theaters sold 19.7 million tickets for Crazy Rich Asians in the United States.
Unfortunately, data on global ticket prices for movies is a lot harder to come by. Each country is different, and many report much less reliable data. This is why box office is usually divided into US vs foreign. For my calculations, given that the film did well in Australia and other wealthy, English-speaking places, my guess is that the average ticket price would come to about $10.00, which means about an additional 6.4 million tickets were sold overseas. If we assume lower, about $7 per ticket, we get to 9.1 million tickets. This brings us to 26.1 million tickets sold worldwide.
That said, in the opposite of Netflix data, where an account could be more than one person, all the “tickets sold” represent less people than the total number. If a person saw Crazy Rich Asians multiple times, each ticket purchased would still count separately. So the 26.1 million tickets is the ceiling of potential viewers.
The Verdict: More People Watched a Netflix Romantic Comedy than Crazy Rich Asians…
I guess I’ll wrap this article up quicker than I thought. The math is pretty clear:
Since 80 million is greater than 26.1 million, more people watched a Netflix “Summer of Love” romantic comedy than Crazy Rich Asians.
The conclusion is even more stark considering that the 26 million is the ceiling for Crazy Rich Asians, while 80 million is the floor for Netflix’ “Summer of Love”. This wasn’t even a competition.
Theaters are dead. Long live streaming.
…Or Did They?
Just because we did some back of the envelope math doesn’t mean we have to stop there. Or, to be blunt, we don’t have to trust Netflix. What if they chose their data selectively? (I trust the box office numbers.)
In this case, we can bring in other measurements to judge the popularity. I saw a ton of articles about Crazy Rich Asians last August, but way fewer about Netflix romantic-comedies. This implies that, at the least, Crazy Rich Asians was really popular. To measure this, we can use Google Trends data to see how often people were searching for Crazy Rich Asians and compare that to the other Netflix romantic comedies released in their summer of love event:
Google Trends data isn’t perfect. Just because something is buzzy doesn’t mean people actually go to see it. This is especially true for theatrical movies: even if you want to see a film, you have to look up show times, find a baby sitter, buy a ticket and drive there. Or you just wait for it to come on Netflix and cable.
That said, Google Trend data is correlated with popularity. Google released a paper in 2013 describing how they could use Google search traffic to forecast box office gross with 90% accuracy about four weeks out from premiere. If you do a quick scan of Google Trend data, well the things you would expect to be popular are. To pick two examples from the August to September time period—when Crazy Rich Asians was at its height—here’s a chart showing Crazy Rich Asians, A Star is Born and The Walking Dead:
(If you’re curious, adding NFL to the chart annihilates them all.)
Google isn’t the only company with tons of user data to use. IMDb (an Amazon-owned company) also tracks millions of visitors checking out actor bios and what not. Again, like Google Trend data, this is correlated with popularity, which is presumably correlated with viewing behavior. IMDb sums this up in to one “popularity” measurement. Netflix itself in its second quarter letter to shareholders even touted the success of “To All The Boys I’ve Loved Before” as reaching the #4 spot in the IMDb “moviemeter” the week after it was released.
On this second measurement of popularity, Crazy Rich Asians tended to perform better than any Netflix original romantic comedy. It peaked at #2 on the IMDb chart (it never could get past The Meg), but stayed high on the list for weeks. Even now, it is #16 on the current iMDB rankings, while no “Summer of Love” movies are still on the popularity list. Using the year to date list for all movies in 2018, Crazy Rich Asians is currently 16th, while “To All the Boys I’ve Loved Before”, which Netflix touted above, is 102nd. (Since this is updating regularly, I pulled the data for this in late November, so it could shift.)
Another measurement on IMDb and other sites is the number of people leaving reviews. Presumably, these are people who saw the movie and decided to leave a note for the world about how they felt. If I wanted to skew the narrative here, I’d leave this measure out, so you know I’m being as honest as I can on my data analysis. By this metric, a Netflix Original over the summer was the most reviewed film, with To All The Boys I’ve Loved Before at 45.4K reviews and Crazy Rich Asians was only at 42K until this week. (Since it released on DVD/VOD, it’s leapt up to 50K and rising.)
If ratings were completely distinct, then this could triangulate to the “80 million customers” number, as all the Netflix reviews added up are about 4.5 times larger than Crazy Rich Asians. That said, people who watch teen rom-coms tend to watch teen rom-coms. And I’m not sure if the older audience of Crazy Rich Asians leaves as many reviews online.
Since each of the different measurements—Google Trends, IMDb moviewmeter and ratings—measure something different, none of them on their own can tell us if more people saw Crazy Rich Asians than a Netflix romantic comedy. Yet, on three different measurements, Crazy Rich Asians was either multiples more popular than all Netflix romantic comedies, or equal to the best. (I almost added in social media, but in my experience it tends to be the most skewed data set of all. Of course, Netflix also told us in their last report they crush it on Instagram. More in future articles.)
So at this point I’d say, we still don’t know what more people watched, but it seems really strange that WAY more people were searching out and reviewing Crazy Rich Asians than Netflix original movies, but that the latter are multiples more popular in terms of actual viewership.
How to Inflate Viewership in the Streaming Age
The solution might be that no one is really lying, but that streaming video numbers can be really, really misleading. With hundreds of metrics on millions of customers that can be sliced thousands of ways, Netflix has a lot of options to present a data story that is extremely positive to them, while omitting all the bad data. (As I wrote yesterday in preparation for today’s big article.)
Here are just a few of the ways a streaming video player could tell a better story. Some of these I’ve seen, some I’ve heard about, and some I just know they could do.
Count Bad Data
Imagine a customer who picks a Netflix romantic comedy, doesn’t like it and turns it off within two minutes. It hardly seems like that counts as “watched”. But under the “customer account” methodology, it could.
Count Wrong Data
A devious way to tell a better story would be to count people who just watched an ad or trailer for a film as people watching the film. This seems like a simple mistake, but I’ve seen this one made before. Databases often use show labels to categorize all material under place, so if you aren’t careful you can make this mistake. Honestly, I wouldn’t be surprised if Netflix did some version of that here, especially if they counted people who auto-played the trailer by hovering over it. This would be extremely misleading.
Omit Key Data
In this example, Netflix only gave us one number so we had nothing to compare it to. In particular, why didn’t Netflix just release the number of hours viewed as well? This should’ve raised a lot of flags for the journalists of the world.
Imagine if Netflix said 80 million customers watched a film, but this resulted in 20 million hours viewed. Well clearly most people didn’t finish a single film! But if the number was 200 million hours viewed, since that means everyone is finishing at least one film, then watching another! Since Netflix declined to tell us either way, I tend to assume the former number is closer to the truth than the latter
Choose the Best Data
Again, with the letter to shareholders, Netflix’s goal is to present the best case to get more investment. This means focusing on the best numbers that support their strategic goals, which is currently investing heavily in Netflix Originals while obscuring their dependency on licensed content. Netflix wants to tout its original content success, so they didn’t give us any numbers of licensed shows like Friends or licensed movies like the Marvel films. Again, they picked the best data to tell their story.
Conclusion: We Don’t Know Which More People Saw Last Summer, but the battle isn’t over
If you’re like me, you’re more confused than when you started. Given the popularity of Crazy Rich Asians, I just can’t believe the Netflix numbers, and yet the numbers are what they are. And the Google Trend data clearly contradicts them. But the ratings is a lot closer. Frankly, we don’t know.
But this battle isn’t over. Crazy Rich Asians was just released on DVD and video-on-demand. It will likely end up on HBO later this year. It could go to cable after that and even Netflix. So it still has a chance to beat all the Netflix original romantic comedies, we just won’t be paying attention by then.
(Oh, and Crazy Rich Asians will make WAY more money, but that’s a topic for the future article as well.)
