My favorite Chuck Klosterman rant is in his book Sex, Drugs and Cocoa Puffs about the phrase “apples to oranges”. In short, is anything actually more similar than apples and oranges? How is that a synonym for difference?
He finishes his rant with the line, “in every meaningful way, they’re virtually identical”. He’s right.
It’s a great line because when doing data analysis, this phrase comes up all the time. When you’re comparing two things, you need to keep as many variables the same as possible or it won’t be “apples-to-apples”. Even a small variable being off can make the conclusions drawn worthless. Ever since I first read Klosterman, I’ve tried to use the phrase “apples-to-hammers” since they truly are different.
The media compares “apples-to-hammers” all the time. Or they’re just bad with numbers. If you want to hear plenty of examples—or just become a better educated news consumer—then you need to listen to WNYC’s On The Media (OTM). Of all the podcasts/shows on entertainment/media/communications, I’d rank it number one, just ahead of KCRW’s The Business.
To take just one example, OTM reviewed a book many years back called Sex, Drugs and Body Counts that describes how the media often over-inflates, or abuses numbers when it comes to wars, crimes or deaths. I have a copy on my bookshelf. The moral of the segment on Sex, Drugs and Body Counts is to beware of a journalist bringing huge, sometimes unbelievable, numbers to tell a sexy narrative. (I can’t find a link to the original segment it was so long ago.)
So, ahem, I need to call out OTM specifically for doing the very thing they regularly decry. Last week, OTM had a great episode on Twitch, “Twitch and Shout” and the future of live-streaming video. They announced this project in their newsletter a few weeks back:
“We wanna tell you about a little experiment that we’re working on here at OTM. Have you heard of Twitch? It’s like the live streaming version of YouTube — if YouTube were obsessed with videos gamers. (It is.)…Well, over 200 million people watch this stuff. That’s more than HBO, Netflix, ESPN, and Hulu combined.”
They quoted a similar fact same number at around minute 13 of last week’s episode.
“It is a streaming network that has more viewers than HBO, ESPN, Netflix, and Hulu combined.”
I bolded those two sections because they sound unbelievable. Twitch is a bigger business than HBO, Netflix, ESPN and Hulu combined. Can you believe it?
Well, I don’t. Because it isn’t true. And because the analogy isn’t apples-to-apples.
Without meaning to, On The Media provided me a great example of how “data”, or more precisely, “an interesting factoid” can be misinterpreted. Today, I’ll break down how OTM was led astray and how we should interrogate data better. Tomorrow, I’ll tackle some other thoughts from the episode and their business implications.
Where the Bad Fact Came From
Let’s start with the fact that OTM didn’t hire a consultancy to measure the number of viewers across all these different platforms. Instead, they likely started with the internet. In this case, OTM found their statistics from a website called, “DOT Esports”, an e-sports news website. Here’s the key quote:
Which service has more viewers, Netflix or Twitch? Turns out it’s the latter. A new report reveals that more people watch online gaming videos than HBO, Netflix, ESPN, and Hulu all combined together.
The “new report” is key. That comes from a company called SuperData Research, also a company specializing in video games and e-sports. So we have to acknowledge right off the bat that both of the sources of this fact are heavily biased towards showing how large and influential their audience is. (No industry body or news source under-hypes its potential.) This is the exact same motivation that was in Sex, Drugs and Body Counts discussed when non-profits or government agencies use big numbers to bolster their own importance.
It seems that after publishing this report—likely accompanied by a press release—this hard to believe fact was repeated on multiple gaming and entertainment websites. Then, these websites were quoted by at least some TV stations. These quotes were found by OTM and repeated without being challenged.
The Bad Fact Itself Isn’t Even True
Reread the quote above and then check out the DOT Esports headline:
“Report shows Twitch audience bigger than HBO’s and Netflix’s”
Note that DOT Esports says that it isn’t that more people watching Twitch then watch HBO or Netflix, but that the total size of the audience of “online gaming videos” is bigger than HBO or Netflix etc.
Indeed, as the chart on DOT Esports shows, saying Twitch has more viewers than HBO, Netflix, ESPN is just…wrong. Here’s my table version of DOTeSports chart from 2016:
Even by the most generous measurement to Twitch, the statement is just false. A combined 325 million people subscribed to one of the four platforms mentioned above; in 2016 Twitch only had 185 million unique visitors. (I haven’t found 2017 unique visitors for Twitch or I’d report that. Given how well Twitch is doing, it’s strange they don’t release this information.)
Of course, in the last few paragraphs I’ve used visitors, people, views, uniques and subscribers interchangeably. And that finally gets us back to the introduction. Even if the Twitch did have more people visiting it then HBO, Netflix, etc, it still wouldn’t be true because the comparison isn’t apples-to-apples.
Apples-to-Hammers Comparison 1: Viewers aren’t views aren’t subscribers
You can see this really clearly in the DOTeSports article when they compare the numbers:
By the year’s end, 185 million people watched gaming videos on Twitch during 2016, with 517 million checking out videos on YouTube. In comparison, HBO had an estimated 130 million subscribers in 2016, with Netflix clocking in at 93 million.
Did you catch the sleight of hand in the above paragraph? The paragraph went from “people watched” Twitch to “subscribers”. Is there a difference? Oh heck yeah. By the time it got to OTM, they changed it to “people” from either viewers or subscribers.
I spent a lot of time at my former employer fighting a losing battle to use terms clearly when it came to our customers. The difference between a stream and a viewer and a unique viewer and what not. This wasn’t an exercise in pedantry; it was vital to the business. I worked really hard so that we didn’t compare things apples-to-hammers and make bad decisions as a result.
So let’s provide a too brief set of definitions. Basically, I’d define the key terms in, roughly, descending order of difficulty to achieve. A view is anytime someone starts watching something. A viewer is the person watching. Does this mean a “viewer” can have multiple “views”? Yes, if they watch multiple videos or the same video multiple times. (So yes, someone could watch a Youtube video multiple times and get multiple views.)
A unique viewer is just charting how many visitors tuned in over a given time period, without counting anyone twice. It is basically saying, “over this time period, we’re only counting this person once, making them unique”. It doesn’t matter if they watch something multiple times, it’s just a “unique viewer”. Even if they only tune in for two seconds, they can still be a “unique viewer” on some websites. This allows websites to measure the number of people showing up on a given day, leading to common metrics like “daily active users” or “monthly active users”. (Though even these can be an average so it depends how you measure it.)
The main problem is that while SuperData Research counted Twitch’s visitors over a year, they aren’t counting ESPN, HBO, Netflix and so on the same way. Way more people watch HBO then subscribe to it; that’s basically a fact. Think about it, do you watch Game of Thrones in a group? Then you would count as, say “five viewers” but only “one subscriber”. This is why this isn’t apples-to-apples. Of course, some people may subscribe to HBO or Netflix and never end up watching, even for an entire year. On the other hand, some kids may watch on their parents accounts without subscribing. On the downside for Twitch, who knows how many ofTwitch’s unique visitors show up for one day and never return? (The latest daily active users for Twitch is 15 million people globally, according to their data.)
Apples-to-Hammers Comparison 2: Subscribers aren’t Unique Visitors, they’re better
That story is also illustrative because being a unique viewer is such a low cost proposition. And again, this is HUGELY important. Twitch and Youtube have “you” as the product, a phrase popularized post-Facebook’s Cambridge Analytica troubles. Twitch sells advertising, so the goal is to get as many total viewers as possible to sell against. (Indeed, they even have a profit motive to inflate all their viewership numbers as much as possible.)
Subscribers are paying per month. That’s incredibly more valuable. So saying one service can get 130 million people around the globe to pay them versus 185 million who may tune in once? Those aren’t nearly as comparable as they seem. Twitch or Youtubedon’t require a credit card to sign up. Just an email address and a log-in. Actually, you don’t even need that to watch the videos, only to comment in the chat room. So again, subscribers aren’t anything like unique viewers or visitors.
Apples-to-Hammers Comparison 3: Geography also matters
This could be the biggest flaw in this analysis.
Twitch does not have geo-filtering meaning its content is available globally, including in China.
ESPN is only available in the US. Hulu is only available in two countries. HBO is available pretty broadly, while also being heavily pirated, and it also has content partnerships, which keep its subscriber count lower, like a partnership with Tencent. Netflix is excluded from China.
It’s worth repeating that last point. Until a recent crackdown in China on live-streaming video, Twitch was available in that billion person country. Netflix has not launched in China because the government won’t let it. A lot of the success of streaming video games comes from other countries.
This isn’t to say that Twitch’s success in China, Korea and Japan (and other countries where gaming is excessively popular) isn’t noteworthy. It definitely is. But it doesn’t really make sense to compare the different services/platform without keeping this variable equal, right? Now, I would love to do a comparison between Twitch and HBO/Netflix/ESPN/Hulu, and for those four companies I could find U.S. subscriber counts, but here’s the fact about Twitch: they don’t release US viewership data.
In fact, one of the weirdest things about Twitch is how finicky it is at presenting it’s own data. If they were super confident in their data, they’d release a ton of it in table form for us to pour over. Instead, Twitch’s advertising site is a selection of data points pulled at seeming random, put next to images that aren’t related. My favorite is the fact that they have “15 million daily active users” put below a picture of the United States. Note, they didn’t say 15 million DAUs in the US, but they want you to think that, don’t they?
Beware the unbelievable factoid; it’s probably not believable
Is Twitch big? Yes. Is it growing? Yes. Is it new and different? Yes.
But just because those questions are true doesn’t make extreme sentences that Twitch is bigger than HBO, Netflix, ESPN and Hulu necessary.
In general, the more a story defies belief, the more we should disbelieve it. Or at least ask where it comes from. The point of a sentence like the one driving this story is to make someone say, “Wow, I know tons of people who subscribe to HBO or Netflix (like myself!) but I’ve never watched Twitch! That must mean a lot of people are watching who I don’t know. I’m out of the loop.”
But you aren’t out of the loop. The statement, “the fact”, is wrong. But I don’t blame OTM too much. They put together a great product every week and some “facts” are so widely distributed it’s hard to believe they aren’t true
