TWO INFORMATION AGGREGATION MECHANISMS FOR PREDICTING THE OPENING WEEKEND BOX OFFICE REVENUES OF FILMS

By David Court, Benjamin Gillen, Jordi McKenzie and Charles R. Plott

Many entertainment and, more generally, cultural products are characterised by significant levels of uncertainty regarding their ultimate appeal to consumers. This is perhaps no place better observed than in the theatrical film industry, where the famous ‘nobody knows anything’ quote of William Goldman has often been used to support this idea. We present two aggregation mechanisms to predict the opening weekend box office success of films and show they are able to provide useful predictions of box office revenues.

We all know that the same set of Hollywood movie studios have more or less been around for 100 years and the industry turns over huge sums of money annually. This leads one to question if the ‘nobody knows’ theory, based on William Goldman, might actually not apply in its most literal sense.  Seemingly, some people do know something.

While forecasting techniques used by economists come in a variety of forms, none of the traditional suite are that well equipped for the idiosyncrasies of the film industry, where the few observable quantifiable variables are rather blunt if they do exist. More commonly, prediction and forecasting in this industry comes down to gut feel of those involved in the decision-making process rather than any serious scientific approach.

With these challenges in mind, one potential way in which we might gain some traction on forecasting revenues comes in the form of an information aggregation mechanism (IAM), a general principle from where ‘prediction markets’ have evolved. The basic premise of an IAM is simple: to collect and aggregate information held by individuals in the form of intuitions, beliefs and opinions using a mechanism that provides incentives for doing so.  Experimental economists have been testing a variety of IAMs in the laboratory over many years and have shown that a properly designed IAM can accurately recover outcome probabilities set by the experimenter using familiar and reasonably simple methods.

One particularly interesting example of an IAM that has been shown to generate accurate predictions derives from the pari-mutuel betting market that is often used in sports betting. The basic idea is that participants place bets over outcomes (e.g. picking the winner in horse racing), and as more money is placed with the bookmaker, the odds change to reflect the underlying probabilities.  After the bookmaker takes their cut, the eventual winner(s) takes their share of the pool in proportion to the amount they have wagered.

In our paper, we investigated whether the pari-mutuel IAM (which we called Boxoffice Prophecy) could make accurate forecasts of box office revenues across a wide selection of films released in Australian cinemas. We recruited industry professionals who were well associated with the Australia film industry as our participants. We provided background information on the films we examined and provided generous prize money to ensure participants were suitably motivated.  The nature of the exercise required bets over revenue outcomes, which were presented as revenue ‘buckets’ as illustrated in Figure 1 below.

Figure 1. Screenshot of Boxoffice Prophecy interface

Once the participants entered the online platform, they were given a budget of token money from which to place their bets. Each exercise was open for one hour and individuals could see where others were placing their bets, and update their own bets accordingly.  After the completion of each exercise, the collective bets placed over outcomes (i.e. revenue buckets) provided a probability distribution that reflected the aggregation of all participants’ collective wisdom.

Of course, the next step to assess whether this aggregated information was of any value required it to be compared to the actual outcomes. A standard statistical test assessed each film’s predicted probability distribution and the percentile at which the actual outcome was observed.  Of course, if the IAM is recovering the correct probability distribution, approximately 10% of the observed outcomes would be expected to fall in the bottom decile of all predicted distributions, another 10% in the next decile, and so on. Figure 2 confirms that, indeed, this is the case as the forecast quantiles follow the actual quantiles almost exactly (visually, the thick black line is statistically indistinguishable from the 45 degree line), although art-house movies were not as well forecast in general.

Figure 2. Kolmogorov-Smirnoff test of all films from Boxoffice Prophecy

We also investigated a very different IAM in the study. The interest in this alternate mechanism was motivated by the difficulty in providing incentives to reveal information today about outcomes that may not be known for some time (if ever).  In our context, we were interested in forecasting box office outcomes of films that were more than six months from theatrical release. We called this IAM ‘Guess of Guesses’ and, as the name nicely suggests, the objective of a participant was to guess the average (actually median) guess of the other participants.  Such an idea is not new to economics and share similarities to the famous ‘beauty contest’ discussed by Keynes.  However, it has not really had serious testing as an IAM in the field.  Simple regression analysis shown in Figure 3 reveals that the ‘fit’ between predicted and actual box office results is reasonably good, with an R-squared of almost 0.7.

Figure 3. Guess of Guesses forecast line fit

The overall results reveal both mechanisms are able to provide useful predictions of box office revenues, although the Boxoffice Prophecy mechanism is found to be superior in our study. The results are of particular interests as they provide evidence that the models tested and validated in the laboratory have application in the field.  From an industry perspective this is particularly useful as it demonstrates that a properly designed IAM can provide accurate predictions by aggregating just a small number of people’s privately-held subjective information.  In cultural industries noted for their high levels of uncertainty, these types of IAMs may prove to be particularly useful.

 

The article is based on:

Court, D., Gillen, B., McKenzie, J. et al. Two information aggregation mechanisms for predicting the opening weekend box office revenues of films: Boxoffice Prophecy and Guess of Guesses. Econ Theory (2018) 65: 25. https://doi.org/10.1007/s00199-017-1036-1.

About the authors:

David Court, Executive Director, Compton School, Sydney, Australia.

Benjamin Gillen, Assistant Professor in Economics, California Institute of Technology (Caltech) Division of Humanities and Social Sciences, USA.

Jordi McKenzie, Associate Professor, Department of Economics, Macquarie University, Sydney, Australia.

Charles R. Plott, Edward S. Harkness Professor of Economics and Political Science, California Institute of Technology (Caltech) Division of Humanities and Social Sciences, USA.

Further reading:

The post is one of the several this blog had on the topic of movies. Another great source about the “movie sphere” is also the website Movie Hustle, which contains a comprehensive list of blogs on movies, including the economics of movies.

Image source:

Foodbeast, https://www.foodbeast.com/news/movie-theaters-fighting-mandate-to-require-calorie-count-on-concession-items/.