Tag: Data

Nate Silver Wasn’t Wrong…We Were: What Hollywood Can Learn from the Election – Most Important Story of the Week 6-Nov-20

Let’s not kid ourselves. This week was about one story. Everyone in America–and around the globe–was watching for the results of Tuesday’s election. I didn’t get any work done on Tuesday or Wednesday because I was distracted by following the news.

So it’s our story of the week!

(As always, sign up for my newsletter to get all my writings and my favorite entertainment business picks from the last 2 weeks or so.)

Most Important Story of the Week – What a Electoral Polling Mistake Says About Decision-Making

I won’t discuss the impact of the election from a political perspective. That is for political pundits, which I am not.

If I’m anything, though, I’m a Nate Silver fanboy. Silver has been incredibly influential for this site. I regularly cite him and his work. I use the phrases “signal and the noise” a lot. And I try to build forecasting models for entertainment. That’s my political affiliation: data truther. 

Which is why the narrative bothers me. Imagine this scenario.

Pennsylvania’s legislature allows votes to be counted several days before the election, whereas Florida votes to end early counting. 

Then, on election night Joe Biden is declared the winner in Pennsylvania. 

Would the narrative have been different? Heck yes.

(To be clear, while Biden leads in Pennsylvania, the race hasn’t been called by networks or the AP nor have final votes been tallied, the latter being the true “count”.)

But the “narrative” is not reality. The narrative is the collective chattering of talking heads, social media conversation, and news coverage. But notably, the narrative is not reality. Reality is reality. See how easily the narrative changes if just two states count their votes differently?

On top of the narrative are the “expectations” that came into the election. These were set, overwhelmingly, by Nate Silver and his 538 model. His model is based on the polls, which had another error like 2016 that underestimated Donald Trump.  (We don’t know the true magnitude yet, which I’ll get to.) This set up a lot of optimistic expectations for Democrats. This set up another narrative, that Silver screwed up his models or that polls are irrevocably broken, or both. Even more than the results, this could be the narrative driving Twitter, “How did 538 lead us so astray?!?!?”

Let me provide just one example from my experience. In my previous role at a streamer, I gathered all the data to help the key decision-makers decide what shows to order and renew. Yet, the data wasn’t mine alone. Often, executives wanted the data immediately. Meaning a streaming show premieres on a Friday morning, and the executives wanted email updates for how the show was performing. Sometimes hourly! Several times, a show would start slow, for whatever reason, and finish strong. Or vice versa. But executives checking every hour would often use their first impression as the takeaway for how the show did.

In other words, I routinely saw the mistakes being made for this election at  a big company in America.

If you’re a decision-maker, your goal is to focus on reality, not the narrative. And where they diverge, you take advantage. If you’re an investor, you invest to make money when the narrative clashes with reality. If you’re in business, you build a competitive strategy off it. And if you’re in politics, you win future elections.

In general, the biggest “error” was not from the polling. The biggest error was how we consumed election night information. (And I’m guilty here too.) Understanding that will help us all–and especially business leaders–make better decisions.

What Nate Silver’s 538 Is Trying to Do (And Other Modelers)

Before we can understand what went wrong, we have to understand what both polls and models of polls are trying to do.

The goal of a poll is to forecast the feelings, opinions or thoughts of potential voters for upcoming elections. It’s a survey! Of course, the goal of this survey is to be accurate. If you could, you’d survey everyone in America. But that would be really expensive! The compromise is to survey a sample and draw conclusions from it. 

The challenge is how pollsters gather that sample. If their sample is biased, the survey won’t be accurate. That is why pollsters get paid the big bucks. Eh, big is too high. Let’s say “some bucks”. Surveys are easy to do, but really hard to do well.

In America, recent technological developments have weakened the ability to survey various populations. Specifically, the rise of cell phones with caller ID has decreased the number of respondents who talk to pollsters. Unlike the past when landlines were ubiquitous, many homes do not have a landline. The potential replacement–internet polls–come with their own sampling problems.

The solution is to adjust the sample population and weight it by various demographic categories. Like age, gender, location, income, past voting history and now educational attainment. Various pollsters use different methods to adjust these results and have done this for decades. Yet, this introduces its own uncertainty. It means that polls are models of what they anticipate the electorate to look like in a given election. 

This brings us to Nate Silver. He makes a model based on polls. Or a “poll of polls”. But since the polls are models, really he makes a “model of models”. The logic is that the average of multiple data points will be more accurate than picking any individual poll by itself. (He’s right here, by a long shot.) And his model crunches tons of additional data and historical evidence to make it as representative of potential outcomes as possible. 

Yet, if all the polls Nate Silver uses have the same correlated error, his model won’t be accurate.

In other words, garbage in, garbage out. But garbage is too strong. So “slightly biased data in”, slightly biased data out. (As it stands, his model predicted 47 of 50 states, but we have to wait to find the margin of error, which is more important.) A model is only as good as the data going into it.

The Models Weren’t Wrong…We Were

Yet, the most misleading thing about the election wasn’t polling errors. The bigger mistakes are still being made.

  • A lot of folks saw the results on Tuesday night, and then rushed out to provide their takes.
  • Worse, even more  folks consumed a lot of data about the election on election night. And then they stopped. (Or they will stop when the election is called.) 

If you’re a voracious news consumer, you’re actually more at risk of this. For example, can you tell me what the polling error was in California in 2016? Did you know that polling actually underestimated Democrats in that election? (If you listen to Nate Silver’s 538 podcast, then yeah, you probably heard him say this, which is where I got it from.)

Yet, because California wasn’t a swing state and folks didn’t check in for final results (which take weeks, unfortunately, to get) most folks never internalized this lesson. They only internalized the miss in Rust Belt swing states. In other words, most folks were not properly informed about what happened politically in 2016 if they focused on election night. Meaning if they had to draw conclusions from 2016, they were more likely to make the wrong conclusions. 

Let’s explain what these decision-making errors are (via their logical fallacies if possible) to correct these mistakes.

Using biased samples/Drawing conclusions too early

This is perhaps the biggest problem with drawing any conclusions from the election:

We don’t have all the data in!

As I write this, California only has about 74% of its vote counted. Many other states are like this as well, and states are still certifying their results. Frankly, you can’t draw conclusions until you have a complete data set, otherwise you risk a biased sample size. 

Which is really ironic, isn’t it? 

The problem with the polls is they have some correlated error which makes them biased…and we judged that on election night with a biased sample size!

Specifically, many urban centers are very slow in counting ballots since they have orders of magnitude more votes to count. Yet, that definitionally makes conclusions biased towards rural communities. This is definitional bias in the “poll” of current vote tallies.

This happens outside of elections. For example, folks evaluate a feature film’s performance on its opening weekend box office. Which is pretty correlated with final performance, as I’ve written. But the two week box office numbers are even more correlated with performance. If you wait two weekends, in other words, you can have a more accurate recall of box office numbers.

The “Temporal data fallacy”/Drawing narratives from sequential data drops.

Obviously, the order of revealing the data shouldn’t impact our conclusions from the data as a whole. What matters are the final results. I’ve taken to calling this the:

The “Temporal Data Fallacy” is drawing a narrative based on the sequential release of data, when the timing of release is uncorrelated with the outcome.

And it doesn’t just happen for elections. In sports, we often weigh what happens at the end of a game much more strongly than what happens in the middle. But a missed basket in the second quarter impacts the outcome just as much as a missed basket at the end, for example.

Availability heuristic/Rare events are easier to recall than common ones.

If you watch all of election night, but have to go back to work the next two nights, then when you recall the election later, you focus on events in the moment, but not the outcomes that happen later. Moreover, the stronger the emotions you feel (like despair at losing) means you recall those events with even more alacrity. Which is why Biden could wind up winning with a greater margin than George Bush in 2004, yet it will feel like he lost because he lost Florida early on Tuesday night.

Folks like to mention Capital in the 21st Century as the most common book that is purchased but not read by intellectuals. I’d offer that Thinking Fast and Slow by Daniel Kahneman as the book that was most read but least applied. We all know the availability heuristic is a thing, but we still are walloped by it on a regular basis.

The curse of small sample sizes/Overconfidence in results.

Elections are a pretty small sample. Which Nate Silver repeatedly tries to tell us, but we usually forget. (They only occur every two years, and Presidential elections every four.)

That’s why his model had everything from a close Biden win to a big Biden blow out in their range of outcomes. With small sample sizes comes greater uncertainty. While Silver has tremendous uncertainty in his model, most folks only focus on the average outcome.

Valuing Process over Results/Expectations

This also relates to the penultimate problem, which is the focus on results over process. Maybe this is philosophical, but I’d rather be wrong for the right reasons, then right for the wrong reasons. The former means I’m still making accurate predictions; the latter means I don’t know. In Nate Silver’s case, his model only gets tested one out of every four years. Sure, sometimes he’s going to miss, but the value is in the model, not the results.

Meanwhile, we often only care about results in terms of the expectations. Thus, Biden will likely win, but since folks thought Democrats would win the Senate too, it feels like a loss. (Winning the Senate, House and Presidency had about the same odds as Hillary winning in 2016, 70-77% in 538’s model.) But the Democrats will likely win an election against an incumbent, which is really, really, really hard to do! That’s worth celebrating, though folks won’t. 

This happens in entertainment even more often. Say two films have the same budget. One is expected to gross $300 million and gets $280. The other is expected to gross $100 and gets $180 million. Sometimes the narrative praises the latter film, even though it made less money. But it beat expectations.

My Advice

So I have recommendations. To make you slightly better at analyzing data in your everyday life and professional role:

– Get rid of dashboards. Dashboards are the election night of data. They take a stream of data in and folks can check them whenever. Even if they don’t need the data or the data is wrong or a decision doesn’t need to be made.

– Determine what numbers are your signal, what numbers are noise, and don’t check the latter. Checking data that isn’t tied to key outcomes will only jumble the narrative and pollute your thinking.

– Check data less often. And check later. This is the hardest thing in the world for decision-makers. A new TV show comes out, so everyone wants to see the ratings. But if you don’t have a reason to check the data–and reason means a decision to make–then checking the data could mean you’re absorbing misleading data. Which the availability heuristic shows will be tough to forget later. 

– Have a “data” plan for your company. (And a communication plan while you’re at it.) This plan should explain what numbers your value and when you check them. And that should be tied to decisions.

– Lower or raise your statistical significance. One of the crazy parts of statistical analysis is how much we still rely on the 5% threshold for statistical significance. This is an artifact of pre-computer calculations. But for some measurements, we need more confidence, and others we actually need less. You should analyze your data with this in mind.

– Ignore most headlines with statistics you haven’t tracked. And please don’t repeat them if you don’t know how they were calculated. Those are likely datecdotes.

Entertainment Implications/Entertainment Strategy Guy Updates to Old Ideas

Entertainment is Filled with Small Sample Sizes

If you take away nothing else, remember this. Much data in entertainment tends to be annual, and that means your sample size is only as big as the number of years in your sample size. In other words, when drawing conclusions, be careful about overconfidence.

(Think box office year over year. Most folks will willingly tout all sorts of reasons for why the box office declined year-over-year or raised, but most likely it’s just statistical noise.)

Surveying Customers Is Still Valuable

One result of the election is folks questioning all political polling. Or asking if this is the end of quantitative data. It isn’t.

 In general, I’m a fan of more data in general: surveys, polling, quantitative, behavioral, even focus groups! They al have a role.

The key is finding which data matters and when it matters. But will “qualitative” data replace quantitative? Hardly. Surveys will still be better than relying on anecdata or datecdotes. 

(In TV in particular, if you get rid of ratings, and rely only on making TV shows using “qualitative” data, that could mean making TV for folks like you. Since you aren’t a representative sample size, this is a bad decision.)

The Presidential Race is Logarithmically Popular

My favorite chart returns! The spending/awareness for Presidential races is orders of magnitude larger than dozens of senate races, hundreds of house races and thousands of state legislatures. That’s logarithmically distributed!

Image 8 - without additionsStreaming Analytics Firms Have Polling Error…But What Is It?

This is a lens I plan to analyze all the streaming analytics companies through, some day. Some firms have potentially biased sample sizes (all users of their service is not representative), others have limited sampling (potential bias), others are limited by their own data (streamers know exactly how many folks watch their content, but not other streamer’s data) and some firms have models with unknown weighting (so you can’t judge the process).  

Given they are the best data we have, I will use their data. Heavily. But I’m aware of its limitations, which lots of news coverage doesn’t seem to be.

4 Insights on Disney’s Content Strategy from the Last Summer

My last few weeks have been spent digging through all the data I could find on the streaming wars. What makes this different than the past is that we finally have a lot of data to parse. Firms like Nielsen, Reelgood, 7 Park and even Netflix themselves have started releasing insights into the streaming wars.

And for the first time, I started to get some insights into Disney’s streaming adventures. Since I was searching for the answer to “How well did Mulan do?”, naturally I found a lot of Disney+ viewership data. And some clear trends emerging about that platform. 

Without further ado, 4 insights on Disney’s streaming content strategy. (By the way, these insights are almost exclusively American since we still don’t have great global data.)

Insight 1: Disney is a Hit Driven Business

In entertainment, you don’t win with doubles and singles. You win with grand slams, since grand slams aren’t worth a bit more, but orders of magnitude more. The top film at the box office earns as much as hundreds of other films, for example.

Streaming hasn’t changed that. Hits are as important as ever. In the last quarter, Disney arguably had the most popular streaming release of the year with Hamilton. Check out Google Trends to see how much more interest there was than any other film in over the last three months:

Image 1 - G Trends with Hamilton

That’s the power of a traditional entertainment studio to find top IP and market it successfully. Going back to launch, arguably Disney+ only succeeded because it launched what is by many metrics the most popular new series in America, The Mandalorian. In other words, in less than a year of existence, Disney launched a show arguably as popular on steaming as Netflix’s top shows (either Stranger Things or The Witcher) and the most popular film of the last quarter. This is a look at just the last week to show how eagerly awaited it still is:

IMAGE 2 - Parrot Analytics Recent Demand

Moreover, as you’d expect, this drives adoption. Here’s Antenna’s sign ups by day chart:

IMAGE 3 - Antenna Longer Time Period copy

No surprise, but big events drive sign-ups. (And the Covid-19 lock down clearly drove signs up in early March, along with Disney releasing Frozen 2 and Onward early.)

Ramifications

The trouble with a hit driven business is you need to keep producing hits. Something Netflix has learned and worked to address in having a big hit each quarter. Disney will need to do the same, and their approach seems two fold: 

– They are building up to a Star Wars or Marvel TV series releasing roughly every quarter.
– Meanwhile, they’ll have their blockbuster films release on roughly a monthly schedule across all their brands likes Disney Animation, Pixar, Star Wars and Marvel. 

Of course, the coronavirus-field production shut downs are mauling this plan. Black Widow was delayed into 2021 and the first Marvel TV series—Falcon and Winter Soldier—due in August still hasn’t had a release date announced. As such, until Disney gets the hits rolling, their new subscriber additions will suffer.

Insight 2: Disney+ Is a Kids Platform First and Foremost

In other words, the vast majority of the viewership on the platform comes from kids watching and rewatching Disney films. To emphasize, rewatching popular content. Look at this chart from 7 Park analytics on the most popular content in Q3, through the second weekend of September:

IMAGE 4 - 7 Park Long Time Period

The shiny object is Hamilton. Again it was a beast. But ignore it.

Instead, look at the next film on the list: Frozen 2. Then 5 and 6: Frozen and Moana.

Yep, Frozen 2 is a juggernaut. Kids don’t just watch it, they rewatch it and rewatch it. But notably, this table is of all audience figures, meaning that the largest majority of customers for Disney+ are families streaming kids content. 

Ramifications

Disney has successfully grabbed grabbed audience share from Netflix in the kid’s space. Arguably, as one of the most trusted brands in entertainment, they had never completely lost it. But instead of letting Netflix build its brand with kids, Disney now owns that part of the relationship. Indeed, in 7 Park’s data, kids content never shows up for Netflix, but it routinely shows up for Disney+ content.

As for the strategy going forward, even Disney will need to refresh its kids content, releasing new films and TV series to keep kids engaged. You’ll also notice this list is all content released this decade. (I assume this is the Aladdin live-action film.) As strong as Disney’s library is, you need to constantly build new franchises. 

Moreover, Disney+ will need those superhero and sci-fi series and films to avoid a reputation as “just” a kids channel. If it is seen as that, that fundamentally limits its global upside.

Insight 3: The Straight-to-Streaming Strategy is Working

You might think I’m talking about Mulan, which would seem to contradict my article from a few weeks back explaining why PVOD didn’t work for Disney. I’m not.

Nor am I talk about straight-to-streaming or quick-to-streaming releases such as Onward (from Pixar), Artemis Foul, or Soul, coming in December. (Also from Pixar.) Those aren’t deliberate choices by Disney, more “best option in a sea of bad options” decisions forced onto them by Covid-19.

Instead, I’m talking about this film in particular…

IMAGE 5 - One and Only Ivan

Again, let’s look at that 7 Park data from above:

IMAGE 4 - 7 Park Long Time Period

The One and Only Ivan held its own against other Disney titles with theatrical releases and in some cases major marketing campaigns. Now, some of this is the impact of marketing the film within the app. Shows and films that get “banner” placement on a streaming app naturally get more clicks. Ivan got lots of that love. And content has been light on Disney+ over the summer, so there were banner spots to be had. Still, looking at the summer as a whole, this film did well.

Ramifications

Well, straight-to-series can work, if you satisfy the number one criteria: keeping budgets in-line with potential SVOD revenue upside.

As well as Ivan did, you can’t attribute lots and lots of customers to it. Instead, this is a solid single that keeps families engaged with Disney+. But it doesn’t drive tons of new acquisition (like Hamilton) or tons of retention (like Frozen/Frozen 2). 

What does this mean? Well, it means you need to have budgets that match that level of demand. In other words, straight-to-streaming video needs to have straight-to-streaming budgets. That means that $150 million budgets are out. $50 million production budgets are out. Even budgets about $25 million are dicey.

Disney both understands this and has experience working in this milieu. High School Musical, The Descendants, and Kim Possible are examples of Disney Channel TV movies, some of which were very successful. I’d add the Hallmark Channel and Lifetime have worked in this budget range for decades. 

The challenge is understanding budget limitations despite the pressure to compete by spending LOTS more money on content. Activist investor Dan Loeb wants Disney to deficit finance to acquire subscribers. Netflix routinely shells out big, big bucks for straight-to-streaming films. And I’ve said they are losing money on some of these flops twice now. First the Irishman and then their big action films.

Image 6 Netflix Hard R Financials copy

Disney needs to invest in streaming without forgetting that theatrical really drives extra revenue. Or they risk losing as much money as Netflix.

Insight 4: Hulu has not Had the Same Success as Disney+

When I talk streaming and Disney, most folks immediately talk about Disney+. And likely it will be Disney’s largest streamer in America soon. But it’s not Disney’s only streamer! Hulu still exists.

When I reviewed all the potential winners of the last two months, Hulu was notoriously absent. I checked in on season 2 of Pen 15 and Woke, but they barely moved the needle compared to Amazon and Hulu’s champions. Here’s my Google look:

Image 7 - G Trends Hulu

Yep, Hulu’s big releases are the nearly flat yellow and green lines on the bottom. This matches my perception via 7 Park’s data too:

Image 8 - 7 Park Summer Data

In other words, Hulu didn’t have a great quarter. Hulu’s best content is still library content or second window shows. Which is fine for retaining customers, but not for adding subscribers. Moreover, Hulu runs the same risk that when deals with big traditional studios run out–like Comcast or CBS–they’ll lose those shows.

Ramifications

Frankly, the fewer hits someone has, the likelier their service is to not be used, which means the higher churn will be. That’s the game in the streaming wars. So take a gander at Reelgood’s comparison of Q2 to Q3 performance by it’s users:

Image 9 - Reelgood

Hulu and Netflix were the services that saw declines; Prime Video and HBO Max saw gains; Disney+ was flat. Hulu is aggressively positioning FX as the brand for that platform. We’ll see if that works, but they need some buzzy shows that drive lots of viewing, and fast. I’d also recommend they focus on crowd pleasing shows—procedurals and sitcoms—which may not win awards or critical plaudits, but that lots of folks watch.

Visual of the Week – The Performance of Netflix Top Films Over Time

(This is a new feature from the Entertainment Strategy Guy. It’s a weekly “visual of the week” that will come out every two weeks. If you like it, consider sharing it on social media, just toss me credit.)

The big Netflix news last week was their earnings report. But the most fascinating story for a data wonk like me was Lucas Shaw’s scoop on the top Netflix films by viewership (2 minutes of a film) of all time. With this scoop, I’m up to 30 different “datecdotes” on Netflix film viewership over time.  

This visual of the week has two different presentations. First, Netflix raw viewership overtime, by quarter:

NFLX visual 3

(Details: This is by my estimates for 70% completion of a film by Netflix subscribers. This is global data. Time period is Q4-2018 to Q2-2020.)

Of course, that doesn’t account for the size of Netflix, so here’s the percentage of viewership:

NFLX visual 2

(Details: This is by my estimates for 70% completion of a film by Netflix subscribers divided by subscribers at the time. This is global data. Time period is Q4-2018 to Q2-2020. Constraint: Only films getting over 20 million subscribers are included.)

If you want more details on Netflix feature film performance, I started a big thread on it on Twitter.