Take a look at this headline (from 2019):
“League Of Legends Gets More Viewers Than the Super Bowl”
Maybe you saw this; maybe you clicked on the link; maybe you even shared it. But if you did, you should know that…
MORE PEOPLE DID NOT WATCH THE LEAGUE OF LEGENDS FINALS THAN THE SUPER BOWL!!!!!
Sorry for the all caps-lock, underline, italicized, bolded text there, but this type of statistic really bugs the you-know-what out of me.
One of my missions with this website is to explain the business of entertainment. To better fulfill that message, I need a few more “explainers”, don’t I? There’s both advanced business terms and concepts that people don’t know—like Porter’s Five Forces, the time value of money—and industry terms—like the definitions of co-productions vs wholly-owned productions—but also some simple terms that people fail to understand or utilize. Today’s post explains the latter, a term I (and many other data journalists) use so regularly that I need one article to explain it:
“Apples-to-Apples”.
In a way, this is probably the least needed “explainer” on the planet. Nearly everyone has heard or used the phrase “apples-to-apples” at some point, so presumably everyone understands what it means.
And yet people make non-apples-to-apples comparisons all the time.
Just look at that headline above. On its face, it’s silly. If more people actually watched the League of Legends finals than the Super Bowl, then esports companies could sell ads for tens of millions of dollars, like the broadcast networks do for the Super Bowl. Is the American advertising industry so moronic they don’t see this huge opportunity? No! This headline is just bad.
What matters though, is why. The comparison isn’t “apples-to-apples”. In fact, it’s a whole series of non-apples-to-apples comparisons:
– The Super Bowl’s viewership is a U.S.-only number; esports’ viewership is global.
– One number is measured by a third-party and the other is collected by the product itself.
– The viewers of the Super Bowl is the “actual minute audience” measured by Nielsen—meaning the number of viewers per minute—whereas the esports number is simply unique viewers.
– Nielsen’s measurement requires at least six minutes of viewing; digital viewing can be measured in fractions of seconds.
(It’s actually not just a bad comparison; it’s also dishonest. Many of the “viewers” are in fact fake. Esports leagues “advertise” on the bottom of certain webpages, and their ads include live esports Twitch streams. Someone who is viewing a webpage, but not even the live-stream itself, gets counted as a “viewer”. If this sounds shady as hell, it is.)
Yes, that headline is from 2019, but just two weeks ago, I got a PR email from a company touting this same type of stat! A few months back, I saw a tweet from a prominent entertainment industry reporter comparing CoComelon’s YouTube views to the Super Bowl.
So folks love comparing things to the Super Bowl in non-apples-to-apples ways!
Today, I’m explaining what “apples-to-apples” means, both why it’s so important but also how understanding this concept can help you be more informed. (Or skeptical of the reporting you read.) I use this phrase constantly—believe me, my researcher went through the website and found over 60 examples—so I want a robust definition and explainer article, for you, to know what I’m talking about. And to share with other people if they don’t understand it.
What Does “Apples-to-Apples” Mean?
Basically, “apples-to-apples” means “to hold all things equal.”
When possible, if you’re comparing two things, you try to keep all the variables the same, except for the one variable you’re measuring. This way, you can assume that the change or difference being measured between the two things is real. In a perfect world, this means accounting for the “5 Ws” of data analysis: who, what, where, when and sometimes why the data was collected.
But it can go well beyond the 5 data Ws. It means taking into account as many different variables as possible. Going back through my archives, I realized that figuring out accurate comparisons is kind of what inspired the Streaming Ratings Report in the first place, since I was trying to make accurate comparisons.
For example, say you want to compare two TV shows, as I do in my streaming ratings report each week. Well, you have to take into account:
– What day of the week a show came out.
– How the show was released, binged or weekly.
– How many subscribers the streamer has.
– How many episodes each TV show has.
– What season the show is on.
– How long each episode is, especially if you’re looking at minutes or hours viewed.
– The genre of the show.
– And so on…
And that’s just to start. Yes, that’s really hard! (Making “apples-to-apples” comparisons is what inspired me to start writing my Streaming Ratings Report.) You’ll never get perfect comparisons—especially compared to media consumption/ratings in the past, when there were a lot less channels and ways to watch TV—but it’s possible to make some. And, in particular, it’s possible to at least not make very misleading comparisons.
Whenever you see a statistic or data-based comparison, the goal is to hold as many variables the same as possible to eliminate bias and confounding variables. As a news consumer, when you read shocking or surprising headlines, your goal should be to read and find how the comparison is (usually) not apples-to-apples.
Why is “Apples-to-Apples” Important?
It reveals truth. Which sounds high falutin’ as hell, but that’s the goal.
If you compare two unlike things, then the results just aren’t valid. We need need to account for all the variables, otherwise our conclusions aren’t accurate.
For example, take this headline published by the podcast/news show On The Media (OTM). They reported that “Twitch is larger than HBO and ESPN…put together!” Does that sound shocking? It should, and that was their point.
But the comparison wasn’t “apples-to-apples”. For example, they compared US subscribers of HBO to global users of Twitch. They compared people who are paying $15 a month for HBO to people who aren’t paying for Twitch. That is obviously not “apples-to-apples”. (Is it ironic a major media outlet that criticizes other media outlets for using biased statistics let this through? Probably. And also a pinch ironic that after I emailed them, they never once responded to my criticism.)
The point is that if you follow the conclusion of OTM, then as a media company you need to invest in Twitch, since it’s much bigger than HBO or ESPN. If you ensure you’re comparing apples-to-apples, though, you wouldn’t end up with that conclusion! Literally better decisions are made by using the data properly.
Bad comparisons/bad data analysis is—and I shouldn’t have to say this—bad.
To be clear: I’m not passing judgement on most people. They just don’t know any better. A lot of people—especially celebrity actors—just repeat any old random statistic that their agent told them about how awesome they are. Reporters aren’t immune either. They often write flashy headlines and hot takes in a rush to publish articles. (Or they’re just confirming their biases and prior beliefs, like all humans do.)
But some bad actors violate the “apples-to-apples” rule to help themselves, especially the streaming companies themselves. (Stay tuned for tomorrow’s post; I’ll have a lot more examples of these bad comparisons in action.) We should call out bad analysis when we see it.
My Approach
I try to be very careful, whenever I make comparisons, especially in my Streaming Ratings Report. I also bring this same level of analysis to my subscriber count estimates. I want to keep things as fair and even as possible. Read here for the eight things I take into account.
Sometimes, if I can’t make good comparisons, I hold off on making these comparisons, because the data isn’t in yet, even if the question/topic is really important. Like one of the biggest questions in entertainment today: should a film go to theaters first or go to a streamer exclusively? We just don’t have a robust enough data set, mainly because so many films, post-Covid, have been released in so many different ways. The comparisons aren’t apples-to-apples.
All of this isn’t easy. Making accurate comparisons takes time and it’s very complicated. And I’m far from perfect. But at least I try. (And I can only do this with the support of subscribers who give me the time to do this level of detailed analysis, lest you end up thinking that a TikTok video was more popular than Squid Game.)
So what “apples-to-apples” means and why it is important. Tomorrow, I highlight the worst comparisons that I’ve seen in entertainment journalism.
