(The EntStrategyGuy is putting up a paywall on 16-June-2022—Subscribe here!—and I’ve been explaining why in a series of articles debunking myths about streaming ratings, explaining my philosophy on data, describing what streaming data we do have, and more. If you’d like to read the rest of those articles and a short case for why, click here.)
Here’s the craziest cognitive dissonance of the streaming wars:
At the same time folks think we have NO streaming data, other folks think we have TOO MUCH streaming data.
To be fair, what many of these people mean is that, while we have a lot of data, we “don’t know what it means”. Everyone is either overwhelmed or confused. Despite having lots and lots of data, no one knows what’s popular because there’s so many different measurement systems and analytics companies.
Worse, some people think that everyone is lying.
Unlike people saying “There are no ratings”—because those ratings exist and most of the people asserting this should know better—I sympathize with people who are skeptical or confused about streaming ratings. Believe me, I get it.
First, a lot of the data that gets released about streaming TV shows and films is really, really vague. I’ve been complaining about this for years, especially when streamers release their own “datecdotes”, my term for vague declarations by streamers that some show or film was their “most watched yet”.
Take this example from Disney+ a few days ago:
That sure sounds good! But we don’t know what that actually means since there’s so many caveats. (“opening weekend”, “Original series”, “globally”) And no—zero, zlich, nada—specific numbers.
Which has been the case for most of the 2010s. If Netflix or Amazon or Hulu said some TV show or movie was their “most watched”, we took them at their word, while mumbling we didn’t know what that meant.
That’s one complaint. The other is that there’s too much data coming from too many people. And I’m sympathetic to this as well. Between Nielsen, Samba TV, TVision, Plum Research, Netflix hours viewed, EDO, Digitali and ten other analytics firms, there’s way too much to keep track of. Plus each streaming analytics company has its own methodologies and metrics. No wonder observers look at it, throw their hands up and say, “What does it all mean?”
Lastly, some people are really, really skeptical about anything the streamers or Nielsen or one of their rivals says. A lot of folks—from online pundits to disgruntled filmmakers and showrunners—think a lot of the data is manipulated. (Or they think these companies are committing outright fraud.) Netflix skeptics don’t trust Netflix and Netflix proponents don’t trust Nielsen. You can’t trust anything anyone says.
Again, I don’t want to hate on this position too much. Both Facebook’s years of inflating video metrics to advertisers and the FCC’s utterly ineffectual response to these lies literally damaged the market, creating understandable mistrust. Here’s an article on Twitch’s misleading data. And another on how half of the internet is fake traffic.
(Side note: Hypothetically, if a hot new social media app, possibly from China, came around and people started getting crazy high engagement numbers, in like 10x or 100x multiples of what they got on Instagram with the exact same content, I’d be really skeptical that those numbers were real and would work really, really, really hard to find a second source of information to corroborate them…)
But, but, but…
This mistrust shouldn’t apply to streaming data, especially when it comes to figuring out what’s popular.
If you’re an entertainment professional whose job depends on knowing what’s a hit, what isn’t, and designing good strategy to compete in a very competitive streaming marketplace, you’re committing malpractice if you don’t know, follow and trust the ratings we have.
Why? I’m glad you asked.
Who Can You Trust? Or, “Know Your Source”
When I say, “trust”, I can feel everyone’s sphincter clench up.
“Trust the media? Don’t you know that they lie?!?!?” If you’re a conservative, you mean the all non-right wing media. If you’re a liberal, you mean big business and conservative media outlets. If you’re a regular person, you mean social media.
One of the most frustrating parts of the recent “discourse” regarding the media and misinformation is that people seem to have forgotten how to analyze sources.
It’s a core part of basic literacy—something I remember learning about in high school in A.P. European History—that it’s a bit shocking this isn’t a part of the regular discourse or people’s media diets. The first thing anyone should do with any source—reporter, pundit, podcast, new website, whatever—is try to figure out who the person is, what their biases are, and so on. We all have biases and that’s okay, as long as we know what they are.
You can apply the same lessons to the sources for streaming ratings.
Should you trust Nielsen? Well, if you’ve trusted them for the past fifty years, there’s no good argument to stop trusting them now. Are they perfect? No way! But when it comes to surveying the landscape, no other single source has as much experience. Should you account for the various biases of their methodologies? Absolutely, but you can also trust them.
Could Netflix be lying? (Since the Netflix stock downfall, I’ve seen at least three former content creators accuse Netflix of “manipulating the data”.) Maybe, though I sincerely doubt it. The risk isn’t worth the reward. If Disney’s brief kerfuffle with ScarJo was a huge scandal and “ruined” Disney’s relationship with talent forever—ignoring ScarJo signing on for another film with Disney, of course—then can you imagine the fallout and lawsuits and permanent damage if Netflix lied about its TV shows’ ratings? Financially and reputationally, it just isn’t worth it.
Does Netflix only/mostly provide data that makes their shows look good? Sure. Do they spin that data? Absolutely, especially in contract negotiations. So we have to take that into account. But they also provide regular, consistent updates…which is the gold standard for data collection.
What about Google trends? Perhaps Google could be manipulating these numbers, but I doubt it. Why would they? There’s no upside for them. As a data source, though, Google Trends and all search data has its own set of issues, especially if shows are “buzzy” beyond their actual viewership impact. And some film titles are either too vague (like “Fresh”) or belong to other groups/people (like “St. Vincent”) which muddles the data. But if a data analysts accounts for that, then I can trust the data.
IMDb? They’re owned by Amazon, so keep that in mind, but it’s not clear that Amazon drives or could drive viewership to their shows by gaming some IMDb numbers. (More likely they just advertise a TON on IMDb.) That said, IMDb ratings are skewed toward genre films and TV shows, so I would keep that in mind too when looking at and analyzing that data.
And so on. You can repeat this exercise with any company that releases information on Netflix. Every streamer, analytics company, analyst, observer, reporter or person has an agenda. Once we account for that, we can use the data.
The Upside? Multiple Sources Allows for Independent Verification
Great, so we can “trust” all this data. (Or figure out how much we trust each data source.)
But do we have too much data?
What? Too much data? Who’d complain about that? The beauty of having multiple sources for information is that you can use them to verify each other!
(I feel, often in these last few years, that I wind up saying blindingly obvious statements…)
Take Netflix, again. About a year and a half after Nielsen started releasing their data publicly, Netflix started releasing global data on all of its shows. (Mainly so confused celebrities can release their own anecdotes after misunderstanding the data their agent probably patiently explained to them while they were half-distracted on their phone.)
And since then, I’ve compared the two. And for the most part, they match up! Meaning the biggest shows on Netflix globally are the biggest films on Nielsen. Is this perfect? No, because Nielsen doesn’t capture mobile viewing and some shows that do well overseas don’t do well in the U.S. and vice versa, but it generally tracks. In fact, I’d be more suspicious if Nielsen’s data perfectly tracked Netflix’s. That would be a sign someone is rigging their numbers.
This applies to other services as well. The top shows on TV Time are also usually the top shows on IMDb which are the top shows on Google Trends and you get the idea. Very rarely does a show miss on the rankings and have great IMDb scores. Or have tons of Google Trends interest. Or high Samba TV numbers.
Finding the nuance between the various ratings is the whole point of my report. But it can be done!
If you don’t trust the data—and I’ve found mistakes in Nielsen’s data, probably more than anyone else—you can verify it! Collecting big data is tough! I don’t envy Nielsen, Samba TV or any other third party’s job. But having multiple competitors—unlike when Nielsen owned the ratings game by itself—allows us to make those comparisons and actually allows for better analysis.
But EntStrategyGuy, How Do We Make Sense of it All?
Well, that’s where I come in.
When it comes to understanding all the different types of streaming ratings, explaining them and understanding the biases, you don’t necessarily need to do all that work…I’ve done it for you. One of my missions is to explain the business of entertainment. That includes streaming ratings.
If you want to be informed long-term, the solution is my weekly streaming ratings report. If you’re the type to throw your hands up and say, “We don’t know what any of this means!” My streaming ratings report is the antidote. I’ll tell you what the data means. I’ll tell you the biases of the data sources as I find them. I’ll tell you the limits of what we can know. I’ll compare multiple sources and tell you what is popular.
My position should be clear: I like having lots of ratings, I trust the sources, but I understand their biases, and I try to combine multiple sources to get a better understanding of the streaming ratings landscape.
And if you subscribe to and read my streaming ratings report, you will too.
