Category: Ideas

The Most Important Shape in Entertainment Part II: Logarithmically Distributed Returns

My dad didn’t like the ending of Empire Strikes Back. His felt that it didn’t finish the story, it left off with a, “See you next movie!” conclusion. That irritated him. He hasn’t seen Avengers: Infinity War yet, so you know he won’t like that.

My article yesterday probably did sort of the same thing to the audience. I come up with this big conclusion—the logarithmic distribution—but then barely touch on it.

Well, since we’re already talking about the movies, we might as use that as the ur-example of my magic trick, “Logarithmically distributed returns”. I first learned this law by analyzing movie performance, and it’s my best tool for teaching it to others. But I’m not just going to show you this phenomena, I’m going to show you it multiple ways, in multiple categories. Then I’ll explain the biggest statistical mistake I’ve seen when forecasting box office performance.

Logarithmically Distributed Returns…What is it?

Let’s start with the last word. What I’m describing today is the “output” of most entertainment or media processes. So my examples are about the “result” or the “y-value” or the “dependent variable”, to describe it in three different statistical terms.

In other words, performance. This means how well something does. Box office for movies. Ratings for TV. Sales for music. Attendance for theme parks. No matter what the format, the success (or very frequent failure) is logarithmically distributed.

What does logarithmically distributed mean? Essentially, orders of magnitude. The returns don’t grow on a geometric scale, they grow on an exponential scale. This means that the highest example can be in the billions while the smallest can be in the dollars. That’s a difference in magnitude of 9 zeroes.

The most common summation of this is the “Pareto principle”, who coined the term about “power law” distribution. Roughly speaking, Pareto is summarized by the 80-20 rule, or 20 percent of the inputs deliver 80% of the returns. And like any mathematics/statistics topic, there are obviously a ton of variations on this law and specifics that I’m not going to get into.

(For those who are curious, inputs have their own distributions, but aren’t as reliably distributed as outputs. A topic for the future.)

Logarithmically Distributed Returns Visualized: Feature Films in 2017

Enough talk about what it is, let’s use an example. I went to Box Office Mojo and pulled all the films from 2017 that grossed greater than $0 in theaters. I didn’t adjust for year and pulled everything, no matter how small. The result was 740 movies released. Oh, and I only pulled domestic gross.

I’m going to show you the data two ways to help you visualize it. First, is the less accurate way, but I love it because it shows scale. This is all 740 movies plotted from lowest to highest, with the y-value as the domestic gross in dollars.


Source: Box Office Mojo.

I love how smooth the curve looks. But the true measure of the data is the “histogram”, where you count the number of examples per category. I set up the categories myself at $25 million dollar in intervals, starting from zero.


Source: Box Office Mojo.

Most people don’t realize how many films are written, produced and even released every year. Like I said, last year was over 700. So let’s add a threshold of $1 million dollars at the box office to our list. If I had production budget estimates, I’d sort by that, but the result gets you to the same place. (The reason for using production budget is that when you scan that “almost grossed $1 million threshold”, you see some legitimate films such as Patti Cake$ and Last Flag Flying, from Fox Searchlight and Lionsgate/Amazon Studios respectively. Those films cost a lot more than $1 million to make.)


Source: Box Office Mojo.

All the charts show the same story in different ways: there are hundreds of films that made less than $1 million at the box office, around 150 that did less than $25 million (many of which probably lost money), a range of movies in the middle and then a few monsters (Star Wars: The Last Jedi, Wonder Woman, Jumanji and Beauty and the Beast).

I think I can hear some of you insisting that I give you the “counting statistics”. You still want to know the average, right? Well here they are, for all 740 films. I mainly did this because I’m going to use them in the next section.


How Logarithmic Distributions Differ from Other Distributions

Perhaps the best way to describe the logarithmic distribution is to show how it isn’t other distributions. In other words, to show how inadequately the normal distribution and uniform distribution capture the performance of feature films.

Let’s start with the uniform distribution. The idea that, “Hey, a movie can gross anywhere between $600 million dollars (Star Wars) and $0, and every where in between.” What if we had an equally likely chance of that? In decision-making, the human brain often defaults to uniform distributions when assessing possibilities, so this isn’t completely academic. Here’s how that would look:slide061.jpg

If only this were how to finance movies! The industry would green light a lot more movies. But it isn’t, only a few films hit that rarefied air of $200 million plus dollars.

What about the normal distribution? I tried to chart this, using our mean of $15 million and standard deviation of $50 million. Unfortunately, that gives us a lot of “sub-zero” grosses, which I just cut off at zero. The problem with the normal distribution is it makes misses as rare as hits. That just isn’t the case. Also, the odds of a giant hit become astronomical in a normal distribution. In this case, a hit like Star Wars: The Last Jedi would be 10+ standard deviations form the mean, meaning it has a 1 in a million chance. Obviously, hits like that happen every year, so more like 1 in 200.slide071-e1536792385695.jpg

Let’s put them all on the same chart, to really show how logarithmic distribution of returns just looks different.slide081.jpg

Source: Box Office Mojo

This chart shows how quickly the results drop off in reality compared to other hypothetical distributions. If someone tells you Hollywood isn’t normal, show them this chart and say, “You’re sure right!”

Variations on the Initial Theme

I might still have skeptics in the crowd.

Maybe, they’d say, I just got lucky. That distributed returns happen to be power-law-based for the year 2017, but this lesson doesn’t really apply to other parts of film. Well, that would be wrong.

Spoiler alert: no matter how you slice the inputs, you get the same result.

First, I could expand the number of years I’m using. I happen to have box office gross from a project I did that covers 2012-2014. Here’s that chart.Slide09

Source: SNL Kagan

Here’s the next fun trick: the distribution of returns still applies for sub-categories. Take horror, which I looked at a couple of months back. Here are all the horror movies going back to the Exorcist, according to Box Office MoJo. Specifically, “Horror-R-rated”, which is 504 films:


Source: Box Office Mojo

The rule still holds! In this case, there has been one monster horror film—It—then some other smaller ones. Of course, I could hold all the box office and adjust them for into 2018 grosses. Does that change the picture? No, if anything it amplifies it. In this case, The Exorcist did $1 billion in adjusted US gross, and The Amityville Horror did $319 million. But for those increases, a lot of other smaller films drop down even more, especially recent films.

slide111.jpgI’ve done this for a ton of different genres. Superhero movies. Foreign films. And it always holds. The only caution is that sometimes the “ceiling” of the range gets compacted.

What about sorting by something else? Say, rating? Do R-movies have more hits versus PG-13 or PG? Fortunately, my 2012-2014 data set has ratings. First, know that G, NC-17 and Not Rater just don’t have a lot of examples (only 45) so I deleted them from this analysis. Here are the other three, in line chart form:slide121.jpg

Source: SNL Kagan

As we can see, for R, it holds. For PG-13, it holds. For PG, it looks like it holds, but honestly since we only have 39 examples, it doesn’t show as clearly. Increase sample size and we’re going to see this.

You could do this analysis setting for production budget and studio and even types of studios. As long as the input is independent, it holds.

Two Examples Where This Works Less Well

Listen, I believe in being up front with my data analysis. Even though this is a magic trick, I’m not trying to hide or obscure data that doesn’t make my case as well. That’s why I left PG rated movies in above, even though it’s the least logarithmic looking line in my analysis.

So in my experience, have I come across sub-sets of movies where my rule/law/observation doesn’t hold? Absolutely, so I’ll share those with you next. To clarify, it’s not that my magic trick fails, it is that the floor disappears. So look at this chart, from my series on Lucasfilm:


Source: Box Office Mojo

These is my data set of “franchises” that included Star Wars, Marvel, DC, X-Men, Harry Potter, Lord of the Rings, Indiana Jones and Transformers. As you can see, those films just don’t have flops. The “floor” is about 200 million in domestic box office, with only 14% of all films dropping below that. So it isn’t logarithmic on one end. I actually think my timeline of films by box office, with their names, shows this floor pretty clearly over time:


Source: Box Office Mojo

My rule doesn’t hold—this is important—when I sort by another output, not by an input. In other words, I’m sorting by the result.

A franchise is a series of films made off a successful first film. In other words, it is sorting by “success” of the first franchise film. Many aspiring franchises therefore didn’t make my data set. Four examples off the top of my head that I did not include, from three different genres: The Golden Compass, Battleship, The Lone Ranger and John Carter from Mars. If I included all aspiring franchises, the list would have looked more exponential Also, this data set is small, only 50 movies.

What about that huge data set I just pulled to look at Oscar grosses? Well, I haven’t even histogrammed that yet, so I don’t know what it looks like. So we’ll see. Again, though, this is in a way a “success” metric in that these are all “good” films. Obviously, a lot of films at the bottom of our list—meaning getting sub $1, $10 and $25 million grosses—were just bad, so no one saw them. With the Academy Awards, we’ve deliberately sorted that out.


Source: Box Office Mojo

The rule holds! Mostly. Now, with adjusted gross we do see a bit of a floor. Historically, a best picture film tended to get more than $50 million in domestic box office. But with both Oscars and Franchise Films, we can see that “super-hits” are still rare, but present.

Final Lesson: This is Why Linear Regression Doesn’t Work in Entertainment.

I have one final lesson for the data heads in the crowd.

Let’s say you’re an aspiring business school student who hopes to go into entertainment. Or you’re a junior financial analyst. Or a statistician diving into entertainment. (Three real world examples I’ve encountered.) You’re given a mess of data on the performance of feature films at the box office. And you want to draw some conclusions.

Well now that we know how our data is distributed—logarithmically—we should come to one clear conclusion: linear regression WILL NOT WORK!

It’s really just right there in the name. Linear regression works on things that have linear growth, and our things have exponential growth, which throws off all conclusions. The work around is that you can convert our data points to logarithms, and then have a “log-normal” distribution, which gets you closer to accuracy. (Though, as I wrote here, you still have a sample size problem.) In general, as well, since you have so few examples of success—the long tail at the right—you just can’t draw statistically meaningful conclusions.

Conclusion – What’s Next?

Well, I didn’t say this was a law of media and entertainment because it applies to feature films. I said it applies to everything. And it does.

But that’s for our next installment and another dozen or so tables and charts!

The Most Important Shape in Entertainment Part I: Distributions Explained!

You want to know a secret? The underlying secret to all media and entertainment? The peak behind the curtain that explains all you see in film, TV, music and more?

Here it is.

“Logarithmically distributed returns.”

Once you learn it you can’t forget it. Like how to do a magic trick, which is what I call it, my magic trick for the business of entertainment. I didn’t discover logarithmic distributions. I first read it in Vogel’s Entertainment Industry Economics, the wonk bible of entertainment financial analysis. (Figure 4.8 in chapter 4 if you’re really curious.) I also assume it’s the theoretical underpinning of Anita Elberse’s Blockbusters, which I haven’t read. (Her book is one of those books that has been on my “to read list” for years.

Unfortunately, I can’t just show you that logarithmic distribution under girds all of entertainment. As important as the “logarithm” part of the statement is, the “distribution” part is even more crucial. I don’t want to gloss over that. The value comes in not just seeing one chart, but seeing the value of distributions as a tool.

Today, I’m going to teach you about distributions. What they are and why you need them. This is a mini-statistics lesson to pair with my other mini-statistics lesson on why you can’t use data to pick TV series. I won’t use any equations, because they’re boring, but I’ll show you what the distributions look like. Then, tomorrow I’ll show you the ubiquity of logarithmic distribution.

(As I recommended before, go pick up The Cartoon Guide to Statistics for the best reader on statistics. Learn them in a weekend. It’s way better than this very useful, but very technical Wikipedia page.)

Before we get to the “what” of distributions, let’s get to the “why”.

We live in a statistical distribution

A lot of news coverage on most issues—politics, sports, criminal justice, business—might mislead you on this point. The world seems like an either or world. This or that. One or the other. Binary choices.

But it isn’t. It’s a distributional world. What that means is that most outcomes fall on a spectrum of possible outcomes. An election could be won by a thousand votes, a million votes or ten million votes. A team can win fifty points, tie or lose by fifty, and everything in between. A blockbuster movie could earn a billion dollars or 100 thousand dollars or anywhere in between. A range of outcomes.

We often try to summarize our distributional life in “averages”. Let’s use an example to make it concrete. Since the NFL season just started, we’ll use that. I found the scoring margin of victories for all NFL games (2,668) going back to 2002 here. (The data set didn’t include ties.) If I calculated the mean average, I’d find that, on mean average, NFL teams won their games by 11.9 points. By median average, that number is 9. Of course, the mode, or most frequent scoring margin is 3 points, followed by 7 and 10.

Those numbers, though, aren’t very helpful. We know something about the data, but in general, we still don’t know what it looks like. Knowing what it looks like is a visual way of interpreting the data’s shape, size and characteristics. That’s where distributions come in. Here’s the above data in chart form:


A distribution, at its core, is a description of data, most frequently using a visualization to show you the percentage of outcomes. You could use tables too, but I’m a visual person. The key is that distributions come in lots of different shapes and sizes. Some fall into similar forms, but many are unique. Those shapes and sizes can have a huge impact on what the data means…impact that is lost in averages.

The Flaw of Averages

At it’s simplest, the flaw of averages is the old saying that a statistician drowned in a river with an average depth of 3 feet. See this cartoon from the San Jose Mercury News:


The coiner of this term, Sam Savageat least who I first heard it fromsums it up on his website thusly:

“Plans based on average conditions are wrong on average.”

Here’s an example of that in action. Say a manufacturing process has a ten steps to it, and each has a 75% percent chance of staying on time. That’s pretty good, seventy-five percent. So, how often is the process delayed? Many would say, “Oh, only 25% percent of the time”. Actually, the result is that the process is almost always delayed! It ends up delayed 94% of the time.

Most businesses, academics and journalists rely on the “average” when it is usually phenomenally misleading. The reason is simple: it’s easy. You have a long column of data, and one excel function returns you the median or mean average. You have to set up an entire chart to show the distribution, and make decisions for how you frame that. If you’re writing to publish on the web quickly, the average is easiest. Often, it’s the sexiest number too.

This has real world consequences. Have you ever seen a five year plan? Of course you have. A five year plan—90% of the time—is a collection of estimates of the average performance of a firm. A CFO took the average revenue projected and subtracted the average costs projected. See where I’m going with this? Financial plans based on averages conditions are wrong, on average.

If you’re reading carefully, you’ll noticed I switched from an example of a data set in the first section—NFL scores—to predictions about future financial performance of firms. This is really the key learning point for distributions: Once we have a description of the real world—be it for sports or finance or entertainment or biology or anything—we can convert our “counts” of real world phenomena into “percentages”. Those percentages become probabilities when we use them to predict the future.

The power of distributions is they help us predict the future, more accurately.

When I write about distributions today and tomorrow, I’ll use data set and probability examples interchangeably. Basically, if you’re describing data in the past, that’s a description of the data. If we use that to forecast the future, we’re in the realm of probabilities. Two sides of the same coin, the past and the future, split by the now.

Since predicting the future is tough—have I written about that yet?—we should use the best tools we have. And averages are poor tools compared to distributions.

Distribution Shape 1: Uniform Distribution

So let’s start with the simplest distribution: uniform. This means that in a scenario every outcome is equally likely. What’s the easiest one to show? Dice!

Quick, what is the average roll of a single dice?

This is one of those brain tricks that I believe Daniel Kahneman and Amos Tversky used to show how behavioral economics works.

Did you say 3? A lot of people do. Take a look at the chart below, showing our first distribution, the uniform distribution:Slide04I’m going to explain the axes so we’re on the same page. The left hand axis, the Y-axis, shows the probability of a specific outcome. The X-axis, the one running on the bottom, shows the potential outcomes. For a six sided dice, you have six outcomes, returning a 1 to 6. If you only had a coin, you have only two outcomes. If you’re playing Dungeons and Dragons and had a ten sided die, you’d have ten outcomes. The more outcomes, the lower the odds in a uniform distribution.Slide05

Mathematically speaking, this is a “discrete distribution” where you have a specific number of possible outcomes. You could also run a uniform distribution as a “continuous distribution”, where it has a range of infinite outcomes. In today’s article, I’m not going to dive deep into the differences between continuous and discrete probabilities, because I mainly want to show the shapes of different distributions, not how to calculate them. I used the dice example above because continuous uniform distributions are hard to find good real world examples. (I went to my statistics textbook on my bookshelf, and it had an example about a pipe bursting, which wasn’t great. Yes, I keep my statistics text book close at hand.)

Back to the brain teaser, most people just naturally think that since three is the two halves of a die (three plus three), it is the expected value of a die roll, not 3.5. Again, the “average” of 3.5 tells you hardly anything about rolling a die; the distribution says everything is equally likely.

Distribution Shape 2: Discrete Probabilities

What if we don’t have a uniform set of probabilities, but a different amount? So we still have a limited (discrete) set of outcomes, but all sorts of different probabilities? To use the dice game, some board games, skew the odds for rolls. So if you roll a six you “win” a prize, if you roll a 3-5 nothing happens, or if you roll a 1 or 2 you “lose”. Scenarios like this happen in certain cooperative or advanced board games like Eldritch Horror. Yes, I’m a nerd who has a stats textbook and plays board games. This outcome would look something like this:Slide06

This type of distribution is great for scenarios where you know all outcomes aren’t equally likely, but you may not have good data so have to make estimates. I did this for my Lucasfilm series in the section of feature film projections. I don’t have data that predicts how many future Star Wars films Lucasfilm will make, but I know all outcomes aren’t equal. Same with box office performance. So I made some assumptions. So here’s how that turned out.Slide07

I converted those percentages to the total box office as a percentage of initial price, which gets us a range of outcomes. (This should look similar to fans of Nate Silver’s 538 website.)Slide08

The key for discrete probabilities is they still need to add up to 100%. Otherwise, you’re missing something. That said, you can quickly complicate it by having correlated variables and other interactions. Again, just know that discrete distributions can look all sorts of different ways, like my Star Wars example or the NFL scores above.

Distribution Shape 3: Binomial Distribution

Regarding uniform distributions, there is really one even simpler than a six-sided dice. It’s the most simple game of chance, and I would have put it first if it didn’t make such a great bridge to the next distribution. That’s the outcome of a single coin flip. In chart form, it looks like this:


A dice is a uniform distribution with two outcomes. Yes or no. Heads or tails. Odds or even. So on. They’re “mutually exclusive” meaning you can’t have them both occur at the same time. The name for this in statistics is “binomial”. Now you can alter binomials in two key ways and ask a lot of fun questions on those alterations: first, you change the percentage from anything above 0 to below 100%. Then, you can repeat the number of “experiments” which is what you call a single coin toss.

What if you take that outcome, and run the scenario multiple times. So you flip the dice twice, or three times or four times and so on? Well, you get a binomial distribution. This a way to show the outcomes of the data and their various probabilities. It looks like this:Slide10

If that looks familiar, well hold on a moment. The key to remember right now is that this type of distribution is the “discrete” scenario where you have a limited number of tests. In the real world, with natural phenomena, you have a continuous range. And that looks different.

Distribution Shape 4: The Normal Distribution

You’ve heard of this one, haven’t you? The chart that shows a peak in the middle, that tapers out to it’s ends? Of course you have. It’s called normal because it is so widely taught, but as I was looking for information, I was reminded that technically this is a “Gaussian” distribution. Here’s from the Wikipedia page that captures the ideal normal distribution.


However, for how common it is, it very rarely occurs perfectly in nature. The classic example is height. Here’s that ur-example:


The funny thing about showing real values is that you can see this isn’t a perfectly even normal distribution. And I pulled this from the US census (and then the link broke on me).

To explain, the x-values, along the bottom axis show the various heights we’ve measured. So we start at five foot four and continue to six feet four inches. The left hand axis, the y-value, is the output which in this case is the count of the sample of people. Or it would be, except in this case it’s already been converted to a percentage to show the population of America.

The results cluster around the middle of the range. So the vast majority of things are close together in around the average. This is why height is such a good explanation for the normal distribution The majority of men are around 5’10 in height, according to the above statistics. And the vast majority fall within 4’ inches of that range, between 5’7 and 6’2. The people who are much, much taller, say 6’8 and above, are very very rare.

The clustering around the mean average is what makes a normal distribution normal. As the Wikipedia example two above shows, 68% of things within one “standard deviation” of the mean. Standard deviation is a measure of how much a data set is spread out, which I probably should have mentioned earlier. In a normal distribution, really rare examples start at “3 standard deviations” from the mean average. So if something is “5 standard deviations” from the mean, like say seven foot tall men, it’s really, really rare.

Of course if somethings isn’t normally distribution, those same conclusions are less important.

Which is a good time to marry the caution I put at the start. I said that I would be using both distributions to show both probabilities and descriptions of data, and height shows how they interact. If you know the historical height of a group of people—and it is statistically significant, which is another stats topic for another day—then you can use the outcomes in the sample group to form probabilities which you can use to predict outcomes.

In other words, given that we know that less than 0.1% of people are greater than seven feet tall in the American population—and we have a sample of hundreds of millions showing this—we know that the odds that any baby born will be seven feet tall are extremely minuscule.

Distribution Shape 5: Variations on the Normal Distribution

The normal distribution can be tweaked in all sorts of ways. First, it can be either very skinny or very wide, as these charts show from Wikipedia.Slide13Second, the distribution can lean one way or the other. It could lean right or left. Here’s two examples of that, again from Wikipedia.Slide14The Most Important Distribution Shape for Entertainment: The Logarithmic Distribution

Well, I hope some of you are still with me. Cause here’s where the magic starts.

The final chart is for distributions that have variance that isn’t linear. It’s exponential. So the numbers at the tops aren’t multiples of sample at the bottom, they’re orders of magnitude larger. I call this, “logarithmic” distribution because it increases by orders of magnitude, most often exprssed in “base 10”.

(I say logarithmically distributed, even though technically that’s for discrete distributions. Also, some power-law distributions can turn into a normal distribution by adjusting the numbers to logarithms. Again, a lot of specifics that I won’t get into. I just want you to see the shape.)

Anyways, take a look at a logarithmic-exponential distribution and a Pareto distribution.Slide16Slide15

To explain one last time. The x-axis running left to right shows the various outcomes. This could be wealth owned. Or the population of cities. Or the value of oil reserves. Or the returns on owning various stocks. The y-axis is the probability of that occurring or the count of the observed phenomena in a sample. So most of people (say 80%) have hardly any wealth. Or most stocks return very little money. But a few at the far right of the distribution have an inordinate amount of wealth. Or a few stocks have incredible returns. (Apple, Amazon). Or have incredibly valuable oil fields (Saudi Arabia).

Or become massive blockbusters at the box office.

Tomorrow, I’m going to show you a bunch of examples in of this distribution, so that hopefully you never use the “average” in entertainment again.

A Quick Addendum to Theme 2: Making it More Complicated

When I wrote “Theme 2: It’s Not Value Capture, It’s Value Creation” last week, I made things seem really simple. Probably too simple.Value Creation ChartThat said, I hold to my core point: most businesses could benefit by pulling out that chart and answering three simple questions,

“What price do we charge customers?”
“What is their willingness to pay (WTP)?”
“What are our costs?”

Then they could ask the forward looking questions: “How can we raise the willingness to pay for customers?” or “How can we lower our costs?” In short, how do we create a competitive advantage derived by creating value, not capturing it?

Real life, unfortunately, is never that simple. That simple chart gets complicated. Really quickly. Here are some ways.

The entire value chain

I kept the chart and examples from the last post relatively simple. I only used one buyer and one seller. But this transaction is repeated down the chain. I pay the store for the beer, the store pays the beer distributor, the beer distributor pays the beer producer and the beer producer pays it’s suppliers of water, hops and aluminum. Each stage has it’s own version of this chart.

Value Chain

This applies to film: the production company pays the talent (who pays a piece to their agent), the studio pays the production company, the distributors pay the studio (theaters, tv networks, streaming platforms) and the distributors collect the money from the customers.

One time transactions versus relationships

Of course, I don’t just go into the story to buy beer once, I go in regularly. (Not that often. Well, maybe.) For customers, regular trips like this can develop habits or a sensitivity to the changes in the price. So I could choose to measure the WTP/Price/Costs as one time events, or over the course of a month, or over a year or even longer. That’s a great way to make something simple complicated.

For example, say you lower the price of a good, which causes a customer to buy it more frequently or larger quantities. In other words, this chart looks like a single transaction, where profits went down, but they would go up with increased iterations. Of course, a customer could just stock up on items and store them, which means you did lose value, but the customer gained in consumer surplus. This is an age old challenge in “consumer packaged goods” that can offer regular discounts.  Like I said, it gets complicated quickly.

This biggest ramification for this for entertainment is evaluating subscription services. Analyzing MoviePass last week, I focused on the per month value chain. Arguably, MoviePass could consider their relationships annually, so they look at it on that basis. Maybe any given month is a bad deal, but over a year it saves you money. Or take HBO, I subscribe for a year, usually, but the biggest TV show by far that I devour is Game of Thrones. Is a year subscription worth that one show? Maybe, so being too lazy to aggressively cancel isn’t that bad of a deal, overall.

Distributions of people

I hate averages. Telling me the average almost never tells me anything useful about a data set. Take height: most men are five foot eight inches tall. Is everyone clustered around that point, or are there outliers? (Maybe an excellent explainer on this next week.)

Same with movie box office grosses. Chart it next to height and they look completely different. One is logarithmic and one is normally distributed.

So the value creation chart is basically the averages, especially for WTP. To extend the beer analogy, some people would pay a lot for a very bitter IPA, other people would pay a little more, many wouldn’t pay anything. And even among the people who would pay for it they have different values attached to the IPA. You can’t really summarize this as one number, though that’s exactly what I did.

When in doubt, use distributions, even with value creation. Understand who gains the most and try to emphasize that, but don’t stop with the averages.

You can’t measure parts of the chain

Especially “willingness to pay”, which is an imaginary value. How do you measure imaginary? Well you have to guess, and there are complicated and often unreliable ways to do that. (The worst way? Ask someone what they would pay for something. That never works.) The most reliable way is a conjoint analysis, but even that can get unwieldy with too small a sample size.

Streaming services are bedeviled by this problem, especially when they have to figure out what consumers actually love on their platform. Is it Stranger Things? Or GLOW? Or both, in some combination? Or is it actually the Disney movies, but the other shows are filler? That’s an epically tough problem to sort out.

Costs can be tricky

The “costs of goods sold” can be difficult to allocate. Especially for support functions that don’t directly tie to a good. Allocating the value correctly can be the difference—in a big conglomeration—between profitability or loss. Right now, content costs and how companies allocate those costs versus the prices customers pay is the biggest accounting/economics/finance question in the industry. Getting that answer right could determine he future of entertainment, for good or ill.

Why Customers Love (Some) Subscriptions with Charts and MoviePass

To put it simply—why not just answer the question in the title early for once?—customers love (some) subscriptions because the consumer surplus is tremendous!

Yesterday’s post really captured why customer hate some companies, so let’s explain the few times when customers love subscriptions. Let’s be clear: in the digital age, when a company decides to lose money, it can be great for customers. Phenomenal even. In business terms, the consumer surplus is huge. That’s right, “consumer surplus” which I introduced on Monday in my article “Theme 2: It’s Not Value Capture, but Value Creation” is a customer’s willingness to pay minus their price. The larger the gap, the better the value.

Let’s use MoviePass as the example of the day to explain the benefits and pitfalls for a company that tries the subscription model.

Why MoviePass?

Mainly, because it simplifies the value creation model to its essence.

First, “willingness to pay” (WTP) is basically a made-up number. Customers don’t really know how much they “would” pay for a good, as that’s not usually how it’s asked. You walk into a store, see a price, and pay it. You don’t usually have a negotiation. Behavioral economics has shown that a lot of pricing is about setting expectations versus a rational cost-benefit analysis. Fortunately, we don’t have that problem with MoviePass. We know the price for a movie ticket, because anyone can just go out and buy one. For this case, we can substitute those prices for WTP. Easy peezy, lemon squeezy.

Second, MoviePass has real costs per transaction, which is that movie ticket from above. One of the big drivers of what I called “digital all-you-can-eat” subscriptions is the low or zero marginal cost of digital products. A DVD needs to be produced in a factory; each additional sale on iTunes has a marginal cost of almost nothing. This can make costs tricky to calculate, amortize or account for. MoviePass doesn’t have that since it’s costs are very clear and very real.

Movie Pass: What is the consumer surplus?

Well, it depends on who you are. In my simplified, single product, value creation model from Monday, the WTP could change per customer, but for the most part everyone is buying one six pack for roughly the same price. With MoviePass, the value per customer depends entirely on usage. Which changes the “consumer surplus” or WTP minus price.

And that single fact explains why subscriptions are either loved or hated.

To show that, we need to make some quick assumptions to illustrate our point. MoviePass—in its epic journey of the last year—changed business plans like ten times. So I’m going to pick what I think was the most popular plan for the longest period of time: seeing an unlimited number of films, once per day, for $10 a month. From Box Office Mojo, I see that an average ticket cost $9 (technically $8.97) in 2017. So we’ll use $9 as the price per movie ticket. (Round numbers, right?)

Knowing this, we can recreate the “value creation” chart from Monday. Let’s imagine four customers, one who forgot to use the service, one who saw one movie, one who saw two movies and one who went hog wild and saw 10 movies.Slide08That can be a bit hard to read, so let’s put it into chart form like Monday.


There is one glaring takeaway from the chart—the fundamental flaw in the MoviePass business model—which is that MoviePass at it’s core is asking a stark proposition: do you use the service or not? If you don’t, like customer one, then you don’t get a value from the product. Even someone who only sees one movie a month would be much better off just buying movie tickets at the theater. On the other hand, if someone goes even twice, they’re clearly getting a better deal than buying tickets from the theaters directly.

What if you’re a “super user”? Going multiple multiple times per month? Well, you’re taking money from MoviePass’ pocket.

The last line on the table shows this trade-off explicitly: MoviePass never created value, it merely exchanged consumer surplus for producer surplus. That’s why you never have the “blue section” (consumer surplus) at the same time as the “green section” (profit) in the chart. MoviePass deliberately couldn’t make money unless a consumer ran a “consumer deficit” versus a surplus. They actively needed customer to sign up for subscriptions and not use them.

That single, obvious fact eluded most coverage of MoviePass.

Taking Our Model and Applying MoviePass’ Real World Numbers

Far from being a bug of MoviePass’ business model, signing up and not using the service was a feature. For this, unfortunately, I have to go to the person running the company, CEO Mitch Lowe, who told the podcast The Indicator that their data showed that the average customer only saw 1.7 movies in a month.

Well, look at where “1.7” puts us on the consumer value chain above. That means he’s acknowledging that at least half of his customer—I’ll be generous and assume the mean is close to median here, though I doubt that—are losing money on the MoviePass subscription. They’d be better off just buying tickets when they go, but instead they’re locked into his long-term contract. (Again, echoes of Columbia House here.)

Months after that above interview, MoviePass had to “pivot” business models. The company was losing lots of money with the unlimited plan, so they changed to a max of three movies per month. Here’s what that looks like in table form:

Slide11Essentially, MoviePass limited their upside risk. They made the “super users” who were using it for a lot of essentially free movie tickets capped to only two free movie tickets. This, though, made the value proposition a lot worse. Thus, when NRG researched this for the Hollywood Reporter, after making the change, MoviePass went form 83% satisfaction down to 48%. In other words, if you take away a lot of free stuff, people like you less.

The “Other Business Models” Arguments for MoviePass

Of course, you could make one of two arguments against my clear value creation chart: what if MoviePass had other ways to generate value for either itself or customers?

The first option is the “What if MoviePass negotiated better deals with the theater chains?”

Well, that wouldn’t really help customers, but would help MoviePass. Assume MoviePass used its size (at one point it was estimated it helped sell 5% of tickets in the US) to negotiate a rate of $8 per ticket. The consumers would get the same benefit, depending on how much they used the service. MoviePass would also lower it’s deficit in most cases. Here’s a chart of that:

Slide10At first, it looks like we might have created some value.

But not so fast. Where did MoviePass get that discount from? From the theaters. To do a true accounting, you’d have to factor in the new deficit to theaters, which would directly equal MoviePass’ negotiation. In other words, all MoviePass did was enter a value chain and demand payment. This is called “rent seeking” and is specifically not value creation.

(MoviePass would defend itself saying it is increasing attendance which drives concessions sales for theaters. Again, theaters could make that trade off—cheaper tickets for higher concessions—themselves without a middle man taking a cut.)

The second explanation is that MoviePass told us it planned to sell our data.

In general, and I hope to get an article published in another outlet on this soon, I’m skeptical of this idea. I’m skeptical of any “secret business plan” that isn’t core to a product. The more obscure the business plan, the less likely it actually exists. Given that data is already plentiful on movie viewing behavior, and the fact that MoviePass didn’t actually sell a lot of data, and given how much money they lost, well this idea wasn’t really real.

The Lessons from MoviePass for Subscriptions

Lesson 1: Customers clearly saw what was a good deal versus a great deal.

MoviePass made it very stark, as people knew the price of tickets in their local theater versus the price of MoviePass. As MoviePass changed/altered/finessed/destroyed their model, customers could immediately do the math to determine if this made sense. As a result, after prices went up and the total number of films available went down, customers saw this subscription wasn’t a good deal.

With many digital “all-you-can-eat” subscriptions, this analysis is a lot harder. Did you watch Netflix last month? Did you listen to Pandora or Spotify or Apple Music? Was it worth listening or watching without ads? Since those questions are a lot more obscure, it makes the decision to cancel that much harder.

Lesson 2: Companies need to price subscriptions very carefully, especially with marginal costs.

This is why most physical goods don’t bother with subscriptions. Just imagine a McDonald’s or fast food chain offering an all you could eat subscription per month. You’d either eat more or less than the cost of the food; if more, you cost the company money; if less, you wasted your own money. It’s very stark with physical goods.

With digital goods or services, the key—especially for non-digital subscriptions—is to price something at the rate that customers perceive they’re getting a good deal, even when you need a majority of them don’t benefit from it.

This is my worry with Lyft or Uber. At my last company, I had an hour plus commute every day. So if Uber offered me an “all you can eat” subscription for lower than my car payment, gasoline and insurance, I would have snatched it up in a hot minute. But for all the Lyft/Uber boosters out there—especially those predicting subscriptions—that number is impossible, unless Lyft and Uber deficit finance it.

My daily commute would cost $50 (at least) for each trip using a ride share. If not more. The true substitution cost assuming 20 commute days a month is $2,000 per month. If it was truly unlimited, I’d use it to go to the store and other places. I know all the people predicting that ride shares will replace car ownership, but they have to explain how long distance commuters won’t devastate the prices of subscriptions. (The answer is the hypothetical “self-driving cars”.)

More likely, the ride share companies won’t offer customers a good deal. They’ll arrange a program that sounds like a good deal, but comes with a lot of strings attached. Once you get caught in the strings, you’ll find that it isn’t that great of a deal.
 Again, Lyft and Uber could keep to per unit pricing with loyalty programs. Once they offer subscriptions, one side of the transaction will likely lose.

Lesson 3: Perception matters more than reality.

Arguably, this is the lesson from Netflix. They price their plan at $11 right now. So every other traditional studio trying to launch a new streaming service immediately runs into a problem: if we mimic Netflix’ price, there is NO WAY we can offer as much content as they do. How do they get away with it? (Hold on a moment.)

Netflix set the perception that tons of content should be available for nearly nothing. Or nothing if you’re using your parent’s account. Unfortunately, if they raised prices to cover costs, that perception may evaporate. So when you launch a subscription, the goal is to show how good a value you offer customers. As shown by MoviePass, this usually means a tremendous consumer surplus. Usually, this means losing money for the company in the near term.

At some point, though, if you can’t cover the costs…

The Main Lesson? Subscriptions that are truly good deals lose money (and Wall Street/VCs pay for it)

I mean, all those lessons above are fine, but the best numbers for MoviePass are the losses it sustained:

$40 million lost in the month of May.

$149 million lost to date from January to April.

It was hemorrhaging money in a way no alternate business plan could hope to rescue it from. But it isn’t the only internet company losing gobs of money in an effort to “secure market share”

$2 billion – Netflix free cash flow losses in 2017 (up from $1.5 billion in losses in 2016)

$900 million – Hulu losses in 2017

$418 million – Spotify operating losses in 2017

I’d also add that Amazon says they have positive free cash flow, but this article by New Constructs says even that might not be true. We don’t know how much money Amazon is making on either Twitch or Amazon Prime Video, though my rule of thumb is if they were making money, they would tell us.

In other words, it isn’t a coincidence that subscription services losing money happen to be the ones customers love. Instead, the more likely explanation is that it is directly tied to offering great consumer surpluses at the price of great producer deficits. Basically Wall Street (and venture capitalists for smaller tech companies) fuel huge producer deficits to enable subscriptions that customers love. At some point, they have to identify a way to actually make money, but that’s a problem for the future, not the near term.

Answering the Question at the Start: Who loves subscriptions?

1. Customers love subscriptions where they have huge consumer surpluses. The only examples of these, though, are where the companies run huge cash losses.

2. Customers hate subscriptions where they have low WTP. The main examples of these are monopolists or near monopolists like cable companies, wireless companies, health insurance or alarm companies. Or they are examples of subscriptions most customers regret after a few months.

3. Wall Street loves both types of subscriptions, for different reasons. They love subscriptions customers hate because again they are near monopolies. They love internet subscriptions because of huge gains in the stock market.

Subscription Business Models Explained: Why Investors Love Them and Customers Hate Them

Here’s a list of companies. Think about how you feel about them:

Netflix. Spotify. Dollar Shave Club.

Here’s another list of companies. Think about how you feel about them:

Comcast. Spectrum (formerly Time Warner Cable). Verizon. AT&T. Sprint.

My guess is you love the first set of companies; you hate the second. What links them? They’re both subscription services.

Subscriptions are hot right now. Hot! The Ringer has their article on subscriptions taking over the world. Here’s an Economist article on subscriptions (they also had a Money Talks episode on thist). Here’s a Forbes interview with the author of Subscribed!. Here’s another book on subscriptions. And since this is a website on entertainment, here’s Variety opining that subscriptions are here to stay.


My quick splashing of cold water on this hot take is this: um, subscriptions have been around in entertainment since the 1980s. At least. If you count “media”, well magazines pioneered subscriptions decades ago, if not centuries, depending on your definitions. (Yep, I just checked and a German magazine was selling subscriptions in 1663.) What was old is new again.

So subscriptions aren’t new, but they may becoming more prominent. I’ve seen two explanations for this: 1. Investors/Wall Street/shareholders loves subscriptions and 2. Customers prefer subscriptions. Are either or both those claims true?

Let’s dig to find out.

Why Investors Love Subscriptions: Customer Lifetime Value

There is a simple economic formula that explains why investors and shareholders love subscription services:

Screen Shot 2018-08-28 at 12.10.27 PM

That’s the formula for “customer lifetime value”, which is the economic way to model the value of a customer in a subscription business. In short, if you know the revenue of a future customer, and the margin you collect of that customer, and the number of people who stay with the program in a given time period, you can calculate the full value of a customer. And yes, I pulled that definition from Wikipedia. Here’s another way to look at it, also ripped from Wikipedia:

Customer lifetime value: The present value of the future cash flows attributed to the customer during his/her entire relationship with the company.

Read More

Be-Twitch-ed: How On The Media Repeated a Bad Statistic and What We Can Learn From It

My favorite Chuck Klosterman rant is in his book Sex, Drugs and Cocoa Puffs about the phrase “apples to oranges”. In short, is anything actually more similar than apples and oranges? How is that a synonym for difference?

He finishes his rant with the line, “in every meaningful way, they’re virtually identical”. He’s right.

It’s a great line because when doing data analysis, this phrase comes up all the time. When you’re comparing two things, you need to keep as many variables the same as possible or it won’t be “apples-to-apples”. Even a small variable being off can make the conclusions drawn worthless. Ever since I first read Klosterman, I’ve tried to use the phrase “apples-to-hammers” since they truly are different.

The media compares “apples-to-hammers” all the time. Or they’re just bad with numbers. If you want to hear plenty of examples—or just become a better educated news consumer—then you need to listen to WNYC’s On The Media (OTM). Of all the podcasts/shows on entertainment/media/communications, I’d rank it number one, just ahead of KCRW’s The Business.

To take just one example, OTM reviewed a book many years back called Sex, Drugs and Body Counts that describes how the media often over-inflates, or abuses numbers when it comes to wars, crimes or deaths. I have a copy on my bookshelf. The moral of the segment on Sex, Drugs and Body Counts is to beware of a journalist bringing huge, sometimes unbelievable, numbers to tell a sexy narrative. (I can’t find a link to the original segment it was so long ago.)

So, ahem, I need to call out OTM specifically for doing the very thing they regularly decry. Last week, OTM had a great episode on Twitch, “Twitch and Shout” and the future of live-streaming video. They announced this project in their newsletter a few weeks back:

“We wanna tell you about a little experiment that we’re working on here at OTM. Have you heard of Twitch? It’s like the live streaming version of YouTube — if YouTube were obsessed with videos gamers. (It is.)…Well, over 200 million people watch this stuff. That’s more than HBO, Netflix, ESPN, and Hulu combined.”

They quoted a similar fact same number at around minute 13 of last week’s episode.

“It is a streaming network that has more viewers than HBO, ESPN, Netflix, and Hulu combined.”

I bolded those two sections because they sound unbelievable. Twitch is a bigger business than HBO, Netflix, ESPN and Hulu combined. Can you believe it?

Well, I don’t. Because it isn’t true. And because the analogy isn’t apples-to-apples.

Without meaning to, On The Media provided me a great example of how “data”, or more precisely, “an interesting factoid” can be misinterpreted. Today, I’ll break down how OTM was led astray and how we should interrogate data better. Tomorrow, I’ll tackle some other thoughts from the episode and their business implications.

Where the Bad Fact Came From

Let’s start with the fact that OTM didn’t hire a consultancy to measure the number of viewers across all these different platforms. Instead, they likely started with the internet. In this case, OTM found their statistics from a website called, “DOT eSports”, an e-sports news website. Here’s the key quote:

Which service has more viewers, Netflix or Twitch? Turns out it’s the latter. A new report reveals that more people watch online gaming videos than HBO, Netflix, ESPN, and Hulu all combined together.

The “new report” is key. That comes from a company called SuperData Research, also a company specializing in video games and e-sports. So we have to acknowledge right off the bat that both of the sources of this fact are heavily biased towards showing how large and influential their audience is. (No industry body or news source under-hypes its potential.) This is the exact same motivation that was in Sex, Drugs and Body Counts discussed when non-profits or government agencies use big numbers to bolster their own importance.

It seems that after publishing this report—likely accompanied by a press release—this hard to believe fact was repeated on multiple gaming and entertainment websites. Then, these websites were quoted by at least some TV stations. These quotes were found by OTM and repeated without being challenged.

The Bad Fact Itself Isn’t Even True

Reread the quote above and then check out the DOT eSports headline:

“Report shows Twitch audience bigger than HBO’s and Netflix’s”

Note that DOT eSports says that it isn’t that more people watching Twitch then watch HBO or Netflix, but that the total size of the audience of “online gaming videos” is bigger than HBO or Netflix etc.

Indeed, as the chart on DOT eSports shows, saying Twitch has more viewers than HBO, Netflix, ESPN is just…wrong. Here’s my table version of DOTeSports chart from 2016:

Twitch Table

Even by the most generous measurement to Twitch, the statement is just false. A combined 325 million people subscribed to one of the four platforms mentioned above; in 2016 Twitch only had 185 million unique visitors. (I haven’t found 2017 unique visitors for Twitch or I’d report that. Given how well Twitch is doing, it’s strange they don’t release this information.)

Of course, in the last few paragraphs I’ve used visitors, people, views, uniques and subscribers interchangeably. And that finally gets us back to the introduction. Even if the Twitch did have more people visiting it then HBO, Netflix, etc, it still wouldn’t be true because the comparison isn’t apples-to-apples.

Apples-to-Hammers Comparison 1: Viewers aren’t views aren’t subscribers

You can see this really clearly in the DOTeSports article when they compare the numbers:

By the year’s end, 185 million people watched gaming videos on Twitch during 2016, with 517 million checking out videos on YouTube. In comparison, HBO had an estimated 130 million subscribers in 2016, with Netflix clocking in at 93 million.

Did you catch the sleight of hand in the above paragraph? The paragraph went from “people watched” Twitch to “subscribers”. Is there a difference? Oh heck yeah. By the time it got to OTM, they changed it to “people” from either viewers or subscribers.

I spent a lot of time at my former employer fighting a losing battle to use terms clearly when it came to our customers. The difference between a stream and a viewer and a unique viewer and what not. This wasn’t an exercise in pedantry; it was vital to the business. I worked really hard so that we didn’t compare things apples-to-hammers and make bad decisions as a result.

So let’s provide a too brief set of definitions. Basically, I’d define the key terms in, roughly, descending order of difficulty to achieve. A view is anytime someone starts watching something. A viewer is the person watching. Does this mean a “viewer” can have multiple “views”? Yes, if they watch multiple videos or the same video multiple times. (So yes, someone could watch a Youtube video multiple times and get multiple views.)

A unique viewer is just charting how many visitors tuned in over a given time period, without counting anyone twice. It is basically saying, “over this time period, we’re only counting this person once, making them unique”. It doesn’t matter if they watch something multiple times, it’s just a “unique viewer”. Even if they only tune in for two seconds, they can still be a “unique viewer” on some websites. This allows websites to measure the number of people showing up on a given day, leading to common metrics like “daily active users” or “monthly active users”. (Though even these can be an average so it depends how you measure it.)

The main problem is that while SuperData Research counted Twitch’s visitors over a year, they aren’t counting ESPN, HBO, Netflix and so on the same way. Way more people watch HBO then subscribe to it; that’s basically a fact. Think about it, do you watch Game of Thrones in a group? Then you would count as, say “five viewers” but only “one subscriber”. This is why this isn’t apples-to-apples. Of course, some people may subscribe to HBO or Netflix and never end up watching, even for an entire year. On the other hand, some kids may watch on their parents accounts without subscribing. On the downside for Twitch, who knows how many ofTwitch’s unique visitors  show up for one day and never return? (The latest daily active users for Twitch is 15 million people globally, according to their data.)

Apples-to-Hammers Comparison 2: Subscribers aren’t Unique Visitors, they’re better

That story is also illustrative because being a unique viewer is such a low cost proposition. And again, this is HUGELY important. Twitch and Youtube have “you” as the product, a phrase popularized post-Facebook’s Cambridge Analytica troubles. Twitch sells advertising, so the goal is to get as many total viewers as possible to sell against. (Indeed, they even have a profit motive to inflate all their viewership numbers as much as possible.)

Subscribers are paying per month. That’s incredibly more valuable. So saying one service can get 130 million people around the globe to pay them versus 185 million who may tune in once? Those aren’t nearly as comparable as they seem. Twitch or Youtubedon’t require a credit card to sign up. Just an email address and a log-in. Actually, you don’t even need that to watch the videos, only to comment in the chat room. So again, subscribers aren’t anything like unique viewers or visitors.

Apples-to-Hammers Comparison 3: Geography also matters

This could be the biggest flaw in this analysis.

Twitch does not have geo-filtering meaning its content is available globally, including in China.

ESPN is only available in the US. 
Hulu is only available in two countries. HBO is available pretty broadly, while also being heavily pirated, and it also has content partnerships, which keep its subscriber count lower, like a partnership with Tencent. Netflix is excluded from China.

It’s worth repeating that last point. Until a recent crackdown in China on live-streaming video, Twitch was available in that billion person country. Netflix has not launched in China because the government won’t let it. A lot of the success of streaming video games comes from other countries.

This isn’t to say that Twitch’s success in China, Korea and Japan (and other countries where gaming is excessively popular) isn’t noteworthy. It definitely is. But it doesn’t really make sense to compare the different services/platform without keeping this variable equal, right? Now, I would love to do a comparison between Twitch and HBO/Netflix/ESPN/Hulu, and for those four companies I could find U.S. subscriber counts, but here’s the fact about Twitch: they don’t release US viewership data.

In fact, one of the weirdest things about Twitch is how finicky it is at presenting it’s own data. If they were super confident in their data, they’d release a ton of it in table form for us to pour over. Instead, Twitch’s advertising site is a selection of data points pulled at seeming random, put next to images that aren’t related. My favorite is the fact that they have “15 million daily active users” put below a picture of the United States. Note, they didn’t say 15 million DAUs in the US, but they want you to think that, don’t they?

Beware the unbelievable factoid; it’s probably not believable

Is Twitch big? Yes. Is it growing? Yes. Is it new and different? Yes.

But just because those questions are true doesn’t make extreme sentences that Twitch is bigger than HBO, Netflix, ESPN and Hulu necessary.

In general, the more a story defies belief, the more we should disbelieve it. Or at least ask where it comes from. The point of a sentence like the one driving this story is to make someone say, “Wow, I know tons of people who subscribe to HBO or Netflix (like myself!) but I’ve never watched Twitch! That must mean a lot of people are watching who I don’t know. I’m out of the loop.”

But you aren’t out of the loop. The statement, “the fact”, is wrong. But I don’t blame OTM too much. They put together a great product every week and some “facts” are so widely distributed it’s hard to believe they aren’t true

Putting Numbers to the Oscar Best Picture and “Popular” Film

If you want to know why I started this website, just take a look at the furor unleashed on the Academy of Motion Pictures Arts and Sciences when it announced (via Twitter!) that it would add a new category called “Achievement in Popular Film”.

First came the questions: “How would this even work? By box office? By user reviews? By top 25 films at the box office?”

Then came the pondering: “Hey, what film would have won in the last ten years? What will happen to Black Panther?!?”

Then came the criticism: “Hey, this won’t work. This won’t solve the problem.” Or summarizing Rob Lowe on Twitter, this will just plain suck.

Throughout all those takes, the data was largely missing from equation. In data’s place lived assumptions. Assumptions from which the rest of the arguments derived. Consider what a flawed world that is: how can we fix something if we don’t know what the problem is? Or worse, when we don’t know what caused the problem?

Well, no more. Let me step into the void with as much data as I can muster to challenge the assumptions permeating the Oscars debate. Let’s separate fact from fiction. Call out our assumptions. Review what we know from what we can only guess. Let’s do this.

But first, my usual warning on data when it comes to media & entertainment.

Warning: We’re in small sample size.

I’m not going to go into as much detail as I did for my series on Mergers & Acquisitions in media and entertainment on my data, but the same admonition that drove that series drives this one: we’re firmly in the realm of small sample size.

Box Office Mojo tracked “Oscar bumps” going back to 1982, so that’s the sample of Best Picture nominees I used. So that’s our starting sample size: 216 films. However, drawing conclusions from 1982 data just seems wrong. Too much has changed since then. From DVDs to going from 200 or so films per year to over 500. As a result, we’ll leverage the last 20 years of data, which is only 136 films, 81 since 2009 and 55 from 1998-2008.

Assumption 1: The Oscars feature fewer and fewer “popular” movies.

Rating: True.

The ostensible reason the Academy needs to make a “popular film” category is because popular films aren’t being included in the nominees for Best Picture. This statement seems obvious which is why so many people said it on podcasts or in articles summarizing the issue. Narratively, this is an easy case to make: In 2017, only two films grossed over $100 million dollars, Dunkirk and Get Out. Worse, the winners in 2016 and 2017 grossed under a $100 million dollars combined, and had a combined box office of $45 million when they were nominated. No films have been nominated since 2014 that we could call a “blockbuster” meaning it did over $250 million at the box office.

(I defined movies as either “popular” with greater than $100 million in domestic box office or “blockbuster” with greater than $250 million. “Popular” and “blockbuster” are my definitions, but they work pretty well.)

The problem with easy narratives is they can often be countered with an equally compelling counter-narrative. If I squint at 2015, it’s hard not to call The Martian a near blockbuster since it did $228 million in domestic box office and more at the international market. I could also point out that La La Land, which was so close to winning it was even announced as the winner, did $400 million in total box office. Or I could just play with the timing: The Oscars don’t have a problem with nominating blockbusters since Star Wars was nominated in 1997, ET in 1982, Beauty and the Beast in 1992, Titanic in 1997 or even Avatar as recently as 2009.

So let’s go to the data. I plotted this a few ways, and they all tell roughly the same story. First, the raw counts:

Slide01Chart 1: Count of “Popular Films” and “Blockbuster Films” in Best Picture Nominees. Data: Box Office Mojo

Even that doesn’t really tell the story right, since the number of films eligible doubled in 2009, as the Academy expanded from 5 films per year to “up to 10”. So here is the percentage of films defined as popular or blockbuster for all films nominated.


Chart 2: Percentage of “Popular Films” and “Blockbuster Films” in Best Picture Nominees. Data: Box Office Mojo

But even that doesn’t tell the whole story. That’s about one or two movies passing a certain threshold. Arguably, the more important fact is the average box office performance of the nominees. How has that trended?


Chart 3: Average Domestic Box Office per Best Picture Nominee. Data: Box Office Mojo

Does even that metric tell a misleading story? See, a dollar in box office in 1998 isn’t equal to a dollar in box office in 2017. According to Box Office Mojo, a 1998 ticket only cost $4.69 whereas on average in 2018 that price has jumped to $9.27. So we need to make the three tables above, but adjust for the price of a ticket in a given year. Fortunately, Box Office Mojo does this for us.


Chart 4: Count of “Popular Films” and “Blockbuster Films” in Best Picture Nominees, in 2018 adjusted dollars. Data: Box Office Mojo


Chart 5: Percentage of “Popular Films” and “Blockbuster Films” in Best Picture Nominees, in 2018 adjusted dollars. Data: Box Office Mojo


Chart 6: Average Domestic Box Office per Best Picture Nominee in 2018 adjusted dollars. Data: Box Office Mojo

Accounting for ticket price inflation, everything looks even worse for the Academy. The worst measure is the average box office per film. It was on a downward slide that was only arrested for a two year period, then it has gone back downward. The number of blockbusters per year looks equally bad, as the period from 1998 to 2004 regularly featured blockbusters, then again besides the period from 2009-2014, they haven’t featured any.

Taking those six charts together, we see a narrative forming that in our time period, we’ve seen two slides away from popular films and towards smaller films. Starting in the 2000s, the popularity of the films started dropping, bottoming out in 2005, and staying in that low period through 2008. So in 2009, the Academy expanded the field to 10 films to hopefully get more “popular” films. It worked, and the number of popular films, blockbusters and average box office jumped right back up.

But after this initial surge of popular films and blockbusters, the voters returned to form and the number of popular films, blockbusters and average box office per film plummeted again. To show this, I combined the data of films through these three time periods:

Slide07Of course, the key question for the Academy is how a lack of popular films reflects in TV ratings. (I’d personally argue that ignoring blockbuster films means the Best Picture category isn’t truly representative of the quality of films in a given year, but I can’t quantify that.) So let’s test that next.

Assumption 2: Featuring more “popular” movies will drive TV ratings for the Oscar telecast.

Rating: Maybe, leaning towards true

Again, narratively this is a really seductive argument. Basically, if you feature really popular films, people will tune in to see those films rewarded with nominations and wins at the telecast. Of course, the counter-narrative is also persuasive and I heard it on two different, influential podcasts (The Ringer’s Press Box? and KCRW’s The Business with Kim Masters): the types of people who watch the Academy Awards don’t watch/like popular movies anyways.

So here’s the one the chart that implicitly everyone referenced but I never saw: TV ratings plotted against average box office per film. (I also did the percentage of popular films, but it was even noisier than this line chart.)


Chart 8: Average Domestic Box Office per Best Picture Nominee in 2018 adjusted dollars versus TV Ratings, in Millions. Data: Box Office Mojo, Nielsen data from Wikipedia.

So what can we draw from this? Honestly, not much. Technically, I should plot this as a regression model using a time series analysis, but I can tell you ahead of time it won’t be statistically significant so I’m not going to introduce that bad data to the world. Instead, the strongest conclusion we can draw is the Oscar telecast peaked in 1998 at 55 million people and have been sliding down ever since.

As for whether popular movies have slowed this decline, you can cherry pick data either way. First, let’s make the “popular movies matter” case. In 2008, ratings hit their nadir at 32 million, just above 2017’s 32.9 million. Each of those years represented near low points in average box office per film. Then in 2009 and 2010 saw increases in the TV ratings, and those two years were the highest average box office since 1998.

I can also make the case that “popular movies don’t matter” pretty easily. 2014 had the highest ratings since 2004—remember this for a minute, I’ll get back to it—and it only featured 1 movie with a box office over $100 million dollars. 2015 saw an increase in the number of popular films and average box office, but ratings still fell from 2014. 2003 had a huge average box office per film, but TV ratings ticked down that year. Moreover, even blockbusters like Titanic weren’t enough to bump up TV ratings, if you look back that far.

How do we draw a conclusion from this? Well, first we admit that one variable like “popularity of films nominated for Best Picture” is just one variable among hundreds. Other variables like the host(s), the date of the telecast, the length of the telecast, the quality of the broadcast, major a-list celebrities nominated and more could all impact TV ratings. Focusing on one variable to explain all our conclusions is a fraught enterprise.

Assumption 3: The Oscars haven’t featured diverse films in the recent past.

Rating: False.

Okay, so I didn’t actually read anyone who wrote this specifically. Or relating it to the move to make “Achievement in Popular Film” a category. But you can’t talk about the Oscars since 2015 without addressing the #OscarsSoWhite controversy.

Arguably, no groups has benefited more from the move from 5 films per year than diverse films and filmmakers. (Except maybe science fiction films, which I’ll get to.) Take a look at the number of films featuring African-American characters or themes measured before and after the expansion in number of films:Slide 9The difference is stark.

In the 9 years since expanding the field, a film featuring African-American characters or themes has been nominated every year except 2015 and 2010, and 2009 featured two, if you count The Blind Side. I debated it since arguably Sandra Bullock is the main character and it is her story, but if we exclude The Blind Side, I’d rule out three of the films from the 2000s for the same reason, which would only make my case stronger. Of the four films in the 11 years before the change that I counted, three are ensembles that don’t really focus on African-American themes. I could easily say that only Ray really qualifies. Before the Academy expanded the field it really ignored African-American characters.

(And obviously I’m only dealing with Best Picture here, not the acting categories, which in a lot of ways was the key driver of the #OscarsSoWhite movement.)

Of course, I could define “diversity” in a variety of ways. Take “global diversity”. Has the expansion of eligible Best Picture nominees helped foreign language films? Not really. From 1998 to 2008, four foreign language films were nominated for Best Picture (Life is Beautiful, Crouching Tiger, Hidden Dragon, Letters from Iwo Jima and Babel) and since then only one film has been nominated (Amour), which is even worse when you consider how many more films are nominated since the expansion.

(I’d also say you could look into Latino-American-themed films, but you’d basically find zero examples of any films in any time period. The Oscars may be so white, but they’re even less Latino-representative than African-American representative. Asian-Americans fair similarly poorly.)

Assumption 4: The Academy wasn’t nominating enough different types of films.

Rating: False, but trending towards true.

One of the biggest topics around this year’s nominees was the bugaboo about Get Out. Specifically, I heard people claiming it was unique because it was a horror movie, and those types of films never get nominated.


To test this, I wanted to see how many “genre” films were nominated before and after this change. I defined “genre” films as any film in the following genres: science fiction, fantasy, war, musical, comedy or animation. And if a film in Box Office Mojo was more a drama than a fantasy (say, The Curious Case of Benjamin Button) or more a drama then a comedy (Juno was a tough edge case here), then I excluded it. Again, we’re looking for films known as genre films, not films that happen to have genre elements.


In this case, nothing much has changed. From 1998-2008, roughly 69% of the films were “drama” films. From 2009 to present, 62% were “drama” films. So it’s gone up by 7%, but that’s within the margin of error. Even the last two years which “felt” like not a lot of genre films were featured still had one-third genre movies.

Of course, for certain categories, the change has really helped. As I mentioned above, since the change 9 clear “science fiction” movies have been nominated: Avatar, District 9, Her, Inception, Gravity, Her, Mad Max, The Martian, Arrival and, last year’s winner, The Shape of Water. Some of those films were also blockbusters or popular. Though the three biggest science fiction films since Avatar (Star Wars) haven’t been nominated. War films have also done very well, but have been generally less popular than the science fiction films, including American Sniper, Hacksaw Ridge and Dunkirk. The other categories all had smaller changes which are more likely noise than genuine signal.

The last category I’d call out is animation. During the first two years when the Academy expanded it’s number of films, two Pixar films achieved Best Picture nominations, Up and Toy Story 3. Arguably, Pixar’s quality the last few years has been at its highest with Inside Out and Coco being frankly, masterpieces (Both were “universal acclaim” on Metacritic.) but it hasn’t seen the respect from the Academy. I would argue that if the Academy really wanted to train a new generation to love movies, putting films like Coco and Inside Out (and even Frozen) would help a lot.

Assumption 6: Politics hurts the Oscars.

Rating: True, but not for the reason you think.

Well, maybe for the reason you think.

The one narrative that was hinted at throughout the #OscarsSoWhite campaign was that featuring non-diverse filmmakers would keep a diversifying America from watching the campaign. As we saw above, though, diverse films have regularly been featured as Best Picture nominees. Instead, the arguably bigger lack of diversity comes from the lack of political diversity in the films.

To explain this, I go back to the biggest discrepancy in the data comparing TV ratings to average box office: why were 2014’s ratings SO high? Again, this year only featured one movie grossing over $100 million dollars, which happened to be its lone blockbuster. So it was an unpopular set of movies that also lacked any major A-list talent. What happened?

American Sniper was the one film.

American Sniper was a major topic on Fox News and other right wing new sites. That’s right, if the Academy is looking honestly at its whole slate of Best Picture nominees, this is pretty much the only film that could be labeled “right leaning” that has been nominated since 2010. (Maybe Zero Dark Thirty too.) The conclusion here is that far from “popular” films, the Academy needs popular films that also appeal to the political-cultural right. (I’m not necessarily recommending that, just acknowledging that this data says.)

Conclusion: A Summary

So here’s my short line summary of the history of the Academy Awards as it relates to popular films and TV ratings.

– By the end of the 1990s, the Oscars featured generally popular films such as Forrest Gump, Titanic, Gladiator and The Sixth Sense, while also featuring critically acclaimed but unpopular films such as Chocolat or The Cider House Rules. 
- Then, from 2000 to 2008, the Oscars featured increasingly fewer popular films, as shown by multiple metrics. The nadir was 2005 to 2008.
– In 2009, the Academy expanded the number of films from 5 to 10 and changed the voting system. As a result, from 2009-2012-ish, the were more popular than the previous five year period.
– Since 2014, the Oscar films have trended downward in popularity, especially among “blockbusters”—films grossing over $250 million—which the Academy hasn’t nominated since 2014.
– The TV ratings have been on a general downward trajectory, though limited evidence (2009, 2010, 2012 and 2014) indicate that popular films can help increase TV ratings.
– The expansion in number of Best Picture nominees helped African-American films/film-makers more than any other category.
– While the films have decreased in popularity, “genre” films have been represented at roughly the same rate. In other words, “dramas”, which tend not to be “popular” have earned about 62-67% of nominations.
– Finally, the biggest discrepancy in the “TV ratings to film popularity” came in 2014, when American Sniper arguably drove the biggest TV ratings in the decade, a film that was the personal favorite of Fox News and its viewers.

I still have a ton of questions to answer on this (some fun, some business) but I think that’s enough for this post.