Often in my weekly column (the “most important story of the week”, click here!), I’ve called out the narratives behind one-off events. Take the Super Bowl. Were ratings down because we’re sick of the Patriots? The death of broadcast? Or because football isn’t popular anymore?
Last year, a lot of people asked, “Why did Solo fail?” and identified four or five totally plausible reasons that all could have mattered or couldn’t have mattered at all. We just don’t know! (I also called out the Lego Movie Part II narrative too.)
If you take nothing else from my website, understand that we need to do better than narratives when it comes to the business of entertainment. (With the implications that execs/companies that rely on data instead of narratives will outperform the others.) One time ratings or box office weekends are noise that we try to force into signal narratives. (Yes, I’m a big Nate Silver fan.)
That said, as fan of film, the Oscars hold a special place for me. I still remember the first film I rooted for at the Oscars and felt devastated that it didn’t win. (Crouching Tiger, Hidden Dragon if you’re curious.) And I’ve seen most Oscar films each year until I had my first child, even if the films I love the most (big, popular and genre) don’t tend to get nominated.
“Popularity” was the meta-narrative of the Oscar’s in 2019, after the Academy announced their intention to start a popular film category. (Well, and diversity.) I first looked at this last August, but now we have the ratings for the telecast on Sunday. Since this year saw a big jump up in box office, without the new category, we can answer the question:
“With a generally more popular set of films, did that boost Oscar telecast ratings?”
The quick answer is that ratings are up (roughly) 12% over last year’s telecast. But what does that mean? Can this one new data point impart the lesson to the Academy that more popular films lead to higher ratings? Not by itself. We need to analyze the larger trends.
Today, my goal is to answer that question, but I’ll be honest, I can’t. The sample size is too small to draw clear conclusions. Instead, my goal today is just to lay out what data we do have and the limits of that that data can explain.
Oh, and to correct the record. I screwed up in August with some data analysis, so I plan to correct the record and explain what went wrong. (With a really fun learning point.)
How to Craft Narratives
First, let me show how easy it is to craft a narrative. Consider these two narratives:
Narrative 1: Popular films boost Oscars.
Obviously, the more popular films that get nominated, the higher the ratings. Is it really any surprise that the highest ratings of the last 10 years came in 2010 (for the 2009 films) when, uh, Avatar and Toy Story 3 were nominated? Meanwhile, the last two years had mostly sub-$50 million films, so ratings sank to their lowest since 2007’s films, which were so unpopular the Academy changed the rules entirely. With the highest box office total since 2010, it’s not a surprise ratings went up 12% this year. Not to mention, Titanic had the highest ratings of all time!
Popular films don’t really impact Oscar ratings.
Actually, it really doesn’t matter. The 2011 ratings were tremendous (10 million more viewers than this year) and the most popular film was The Help at $169 million. Or 2005. The biggest nominee that year was Brokeback Mountain (that’s a fun trivia question to stump your friends) with $83 million. And 38 million people tuned in. Sure popular films may matter, but even a juggernaut like American Sniper didn’t help boost the ratings, as they declined from the year prior. (It did way more box office than Bohemian Rhapsody or A Star is Born.) So yeah, if you care so much about Titanic and Avatar, maybe you just need an awards show devoted to James Cameron movies, and leave this awards show alone.
Why is it so easy to craft narratives? A small data set
Narratives don’t help. Instead, we need data. But data alone can’t solve our problems, and I’ll explain why.
The Explanation 1 – Small Sample Sizes
In the realm of small sample size, everything can be true. Simultaneously.
I weaved those paragraphs above by looking at my Oscar film table and picking high and low years, while cherrypicking the data. With annual data sets, you only get one piece of data each year, the Oscar’s telecast.
Further, this data set is limited by history. I can’t justify including years before 1998, since that was a time period when broadcast shows like Seinfeld got ratings in the 20s. Since then—and even before—cable has been taking viewers. (Hot take: the biggest driver in the decline in broadcast ratings over the last 25 years has been cable television, not streaming.)
Then in the middle of that data set, the Oscars expanded from 5 films to 10, then somewhere between 8 and 10 since. That means even my 20ish sample size is arguably only 10. And yeah, cord cutting started in the middle of that latter ten years. So a five year sample set? That’s small.
The Explanation 2 – What are we measuring for?
This seems easy—popularity!—but is deceptive. Do you take viewers in millions—which is growing?—or ratings—which is declining? Or growth per year? Or rolling averages?
Or take the biggest “input” being popularity. Obviously, box office is the best measure for popularity, because paying to see something is the truest expression of intent. But how do we measure those 5 to 10 films?
This year, a lot of people just added up al the box office numbers to get the total box office. But clearly years with 10 films have an advantage over years that only have 8 (or 7 if one was on a streaming platform). So you could use the average to account for that, but then again one huge outlier (Avatar in 2009 or Black Panther in 2018) would throw that off. Or maybe not, since I’ve always said this is an industry dominated by logarithmic returns and the outlier could draw in more viewers.
Still, if you did want to account for the number of films appealing to the most people, you could factor in box office ranks or median box office or the number of popular or blockbuster films. All of which I did in this table (which has been updated to the last weekend of box office):
The point is, I came up with 16 different ways to even ask, “is this set of films popular?” That’s partly why conflicting narratives can arise.
The Explanation 3 – So many variables
Finally, the last difficulty is that beyond popularity, quite a few variables can and do impact the ultimate TV ratings for the Oscar telecast. Off the top of my head:
– Presence of big stars in feature films
– Decline of broadcast TV ratings, in general
– Decline of broadcast TV ratings, because of cord cutting specifically
– Popularity of the host
– Quality of broadcast the previous year
– Politicization of the Oscars (cuts both ways)
– Popular films actually “contending” for Best Picture, not just nominated
And likely more. So a small sample set, with many ways to measure our variables, and a lot of potential explanations, of which most we can’t test.
Trying to Answer the Question
C’mon Entertainment Strategy Guy. Do or do not, there is no try. So here’s my try.
Step 1: What is the null hypothesis? What is our hypothesis?