A few months ago I briefly tried to explain the distinction between “customers” and “views” to help explain why Twitch is often over-hyped. Since I’ve spent a lot of the last two weeks banging my head against the Twitter wall insisting that we stop letting Netflix use misleading data, it seems time to break out that explanation into its own post.
To see the need for this, let’s look at a handful of recent Netflix announcements. They provide a case study for how a service can use multiple metrics that all kind of mean the same thing but all don’t. Worse, a lot of the journalism covering these reports mix up the different words. In 2018, at some point, Netflix has said…
– 80 million customer accounts watched a type of movie.
– Customers watched 20 million streams of a single movie.
– 80 million customers watched a type of movie 300 million times.
– In one day, Netflix had a total of 35 million hours viewed.
In those four datecdotes, we have, really, three different concepts: streams, hours and customers. The key is understanding how they all interact so we don’t use them haphazardly or misleadingly. If this explanation comes across as obvious, well apologies in advance. But as I think about it, I didn’t know it before I worked at a streaming video company, did I? Nope, and I spent a lot to time explaining to senior leadership what our numbers did and didn’t mean.
So let’s get started at the smallest level.
The Starting Point: An entry in a database
To understand where all the streaming numbers come from, you first have to understand that every data point for a streaming video company comes from somewhere. That somewhere is a single entry in a database.
Yeah, it seems obvious, but worth mentioning. There is a database that holds the record of every customer’s every interaction with Netflix, Hulu, Amazon, CBS All-Access, Showtime, DC Universe and Youtube. And any new streaming service down the road. That’s where all the data comes from. A massive database that tracks every interaction.
The key is that lowest level, “the interaction”. The specific details around the record will differ by company and for different reasons. But the general broad strokes are the same. These interactions are then complied and collated and analyzed to develop all the other advanced metrics.
A Sample Entry Explained
The best way to visualize an interaction is to see a sample. So let’s see what a sample database entry looks like. This way you can understand the specific pieces of knowledge the companies can track. It starts with the “Five W’s” (who, what, when, where) and builds out from there. (The “why” is the key to good decision-making, and simple statistics can’t tell you that.)
An entry is generated when you—the user—clicks on a show or movie to watch on a streaming platform. That something can be a movie, TV show, trailer, commercial or whatever. Or piece of music for a streaming service. But the click via mouse click, remote control tap, voice command or finger tap starts the process.
Let’s just go through each piece. Start with the “who”. Every customer is tracked by some sort of customer ID number. This means that it tracks everything related to one account. I called this a “customer”, but you could call it users or customer accounts. Notably, it could be different than a “profile”, which Netflix has. (And if you have a “kids” section, then you are subject to COPPA regulations, and shouldn’t track identifying data, a different issue.)
Next, you have the when. Specifically, the time the interaction started. This is stored down to the second at least, usually beyond. It therefore captures the year, month, day, and hour. This allows you to sort by time periods. (Meaning you can limit the time period to a daily, weekly, or monthly report.)
I’ve seen different ways to track the next part, which is “how long you watch for”. One simple way to think about it is that the service tracks when you start and stop you session. The interaction. If you subtract the start from the stop time, you get the minutes watched. This means we know how long the interaction took place. You could either have the database track this separately, or run the calculation on your own later.
Then we get to the what. This is what specific type of content you watched. Down to the episode, if necessary. Now, companies have a lot of ways to categorize their content libraries, but the point is the streaming company knows exactly what you watched, usually categorized by content ID number or unique title. These ID numbers for content could be categorized into seasons, series, TV versus film or genres, depending on how the streaming provider wants to sort its data. Or it can have a separate database with that information that is linked later in the analysis process.
I’ll add, you can get even smaller in the episode, which is to know what part of a piece of content you are watching. So you can say start from the beginning, minute zero, and watch through minute 22, of a TV series. Or you can watch just through minute five. That level of detail can be tracked as well. This allows the service to remember where you left off for the user experience.
Companies also know the “where”, usually by IP address. Since you’re accessing through the internet from somewhere, the IP address gives the location. (The country level data is probably tied to the customer account, though, if credit cards are involved.)
Everything else is bonus stuff. What type of device is watched is tracked. (And usually categorized between mobile, living room and computer.) What type of customer you are is tracked (free trials or paid subscriptions). What type of content, movies versus TV versus marketing material. Who your provider is related to device, so did Sony or Apple send you to the platform? It matters. And I’m probably forgetting some obvious ones.
(Oh if credit card are attached to an account, that can provide a wealth of additional personal data to pair to the viewing behavior. Like content type, this is usually married into the data later.)
Some Definitions – The Simple
So every piece of data revealed to us by Netflix or Hulu or Amazon Prime/Video/Studios is a summary at the highest levels of that viewing behavior. With millions of customers, this is big data we’re talking about and it requires some careful—but now common place—analysis to do right. But the definitions are the most important part, because a lot of this gets summarized as “viewership” which is not the same thing.
Customers
Also called: Users, Viewers, Uniques
What it is: A count of all the unique customer IDs watching a given piece of content. Basically, this counts the people doing the watching. And the easiest way to do that is at the account level. Since every customer has a unique identifier, especially for subscription services, this is easy to track.
What it is NOT: The key challenge here is that customers does NOT equal viewers. If I sit down with my daughter to watch a Disney movie, we’re only counted as one “customer”, when there are two of us. In traditional TV, metrics, Nielsen is specifically trying to factor in how many people are watching in a household, since that makes more sense for advertisers. Not so for streaming. If Netflix had 80 million customer accounts watch a show, we have no idea how many people that actually tuned in, but we know it was greater than 80 million. (Not that they don’t try to know, it’s just tougher with this dataset.)
The other confusion is when people mistake the next metric, streams, for customers.
Streams
Also called: Views. Watches.
What it is: Simply a count of all the interactions. Add up all the people of any type who clicked on a piece of content, and count each click separately. (Sometimes you have a time limit of say 3 or 5 or 30 seconds before it counts.)
What it is NOT Part 1: The main reason I am writing this article now is because Ted Sarandos said that The Christmas Chronicles was “streamed 20 million times”. What did he mean by streams? Well, he simply meant that they had 20 million entries in the database. If he had been pushed to clarify, he would have said that this counts every time someone started a stream. If it took you four tries to watch a film—say you had interruptions for dinner—that means you got four streams. That’s why streams are some of the most misleading statistics. A stream is really just an entry, no matter how many times the entry came from the same customer.
What it is NOT Part 2: Streams is NOT completed streams, which is the biggest misdirection by Netflix, Amazon, Hulu and even Youtube when discussing either views or streams. Now, Youtube is both clear and unclear on this point. They will tell you that if you watch 3 seconds of something, that counts as a “view”.
But honestly, most people think if a streaming video company says someone “watched something” that means all the way through. Usually, it takes multiple streams to watch something. For example, if your computer crashes and restarts, even though it is nearly the same time, that counts as two streams. This fuzziness in definitions is why I prefer the next category.
Hours
Also called: Minutes. Hours viewed.
What it is: Add up all the seconds into minutes, all the minutes into hours and you have the “total hours viewed” of any given episode, season, series or movie.
What it is NOT: Whether or not a viewer finished a show or movie. Say for some reason the same customer watched the first part of a movie over and over. I don’t know why they would do that, but say they do. Well, the raw hours number doesn’t distinguish between that or someone who watched the entire movie. Hours just counts the total minutes/hours you watched a specific piece of content. It also does not care how many tries it took or how many streams, it just adds all the minutes of viewing together.
An Example – Completion Rate
So let’s take something more complicated, and show how you can take the pieces above to generate it. Say you do want to know “completion rate” of a movie. How would you do that?
One idea would be to just take the hours watched. So take say the 80% of total view time of each movie, and have the database only pull people who’s total minutes viewed exceeded that. I’ve seen this done. (Or you pull all the customer data with hours viewed, and sort it later.)
Of course, for weird reasons, maybe someone just rewatched the opening song—say for a musical—so you change it to only people who watched past the 80% point of a movie or TV show. But that could still include people who, for whatever reason, skipped to the end of a movie just to see that. I don’t know why they would do that, but they could. (And with millions of interactions, you’d be surprised what weird stuff you see.) So you could do both metrics to be extra sure. (By definition, including both 80% of total minutes and viewership past the 80% point would be less than either one on their own.)
These pulls require a higher level of coding in the database to do accurately and well. But they aren’t exotic by any means and are often automated. For TV series, you could usually skip the minutes count and just track if a single customer ID watched each episode in a series, especially if the series isn’t a procedural. So if a customer watched all of three episodes, but didn’t start episode 4, you assumed they didn’t complete the season.
(Though again, you’d be surprised how many people start some TV seasons in the middle, even when they are exclusive to a platform!)
Using Our Data Examples Above
So we’re now armed with how the database looks and what the pieces mean. This means we can use an example to see how Netflix pulled the data for one of their shows. Crucially, we can see that they probably pulled the exact piece of information we would have loved to seen, but then left it out of the report.
So for the Netflix “Summer of Love”, it likely pulled the streams, hours, and customers metrics for each show, with a completion rate. And probably a bunch of other data. But let’s stick to one simple table for now. Here’s a summary of the Netflix “Summer of Love”, which was eleven films:
Crucially, this table is a worldwide table, meaning it adds up all viewing. You could remake this table by country or region. As you can see, there are 49 potential data points, but Netflix picked one.
You can see this as well with The Christmas Chronicles. Netflix had a simpler table in that regard as they only needed one line. Like this:
This makes Ted Sarandos’ decision pretty stark. He looked at four major numbers, and picked the highest sounding one, which happened to be streams. And happened to round to “20 million” which is a great sound byt. Not customers, not hours, not hours per customers, but streams.
The Caution: Don’t cross the streams
Unfortunately, in reporting on Ted Sarandos—who deliberately picked the word “streams”—a lot of Twitter and journalism converted streams to customers. Or even “people watched”. (Here’s The AV Club making that mistake.) As you now know, these two things are definitely not the same. But Netflix knows that this crucial distinction is often lost as their message is distributed.
Now that you know how the data works, you can appreciate that customers does not equal views and streams aren’t the same thing as hours. So in the future, don’t cross the streams and hours and customers.
