3.1. The Mistreatment of Uncertainty

It is difficult to specifically critique commercially available pandemic catastrophe models because they are proprietary. However, the documentation for several hurricane catastrophe models is available for public inspection as a requirement of the Florida Commission on Hurricane Loss Projection Methodology. Given the basic similarities of catastrophe models, some general remarks can be made regarding their attempts to characterize uncertainty by aggregating scenarios through a large number of stochastic simulations. In these cases, the numbers of simulations or aggregation methods are not the issue, but the method of representing uncertainty.

There is no current consensus on the best way to represent uncertainty or when to use one form over another [24]. Furthermore, there is no consensus on the number of forms of uncertainty. Uncertainty has been categorized across a broad spectrum that ranges from certainty to nescience (unresolvable ignorance) [25]. Bayesians [26,27] generally argue that there is only one type of uncertainty—a measure of belief irrespective of its source—and that probability is the best way to express it [28]. At the other extreme, complex uncertainty typologies have been created that distinguish it by location, nature, and level with multiple subtypes [29,30]. The utility of this level of detail is debatable when there is no method for treating each distinct form of uncertainty [31,32].

While uncertainty representation is an unsettled matter, distinguishing between incertitude (i.e., lack of knowledge) and natural variability does have known advantages over a single treatment methodology. For example, when an analyst has no knowledge of the value of a parameter within a bounded range, it is common to represent that interval using a uniform distribution (a frequently used “uninformed prior” in Bayesian inference). However, treating interval data as uniform distributions requires an equiprobability assumption—an idea that traces back to Bernoulli’s and Laplace’s ‘principle of insufficient reason’ and more recently critiqued by Keynes under the name ‘principle of indifference’ [33]. Under this assumption, each possible value is considered to be equally likely and thus the interval can be represented by the interval’s midpoint—the mean and median of a uniform distribution. While the uniform distribution approach is computationally expedient and easy to understand, it can also disguise uncertainty. One result of the Central Limit Theory is that any distribution with finite variance, such as a uniform distribution, will converge to a normal distribution approximation when repeatedly convolved [34]. As shown in Figure 1, adding two uniform distributions results in a triangle distribution that would appear to have more certainty than the original uniform distributions. As the number of distributions increases, the resulting normal approximation does spread, but the assumed central tendency is maintained. By comparison, the addition of two intervals of [0, 1] by interval arithmetic yields the even wider interval [0, 2]. Repeating the process yields ever wider and more uncertain intervals without an artifact of precision.

Figure 1

Convolution of n uniform distributions with bounds [0, 1].

The practical implication of this phenomenon is that standard Bayesian and Monte Carlo techniques have an important limitation. These approaches will yield valid results when the model inputs represent normally distributed natural variability. However, when the uncertainty due to lack of knowledge is large compared to the natural variability, the uncertainty in the model output will be under-represented. Unfortunately, this situation frequently occurs in rare natural disasters, such as pandemics.

In practice, uncertainty representation may be based on extraneous factors such as familiarity, academic tradition, or lack of knowledge of alternatives. Catastrophe modelers have been warned in the past to avoid becoming enamored with extreme precision that may not have a basis in reality [35], but they need further guidance. Fortunately, there has been considerable work in the area of intervalized probabilities [36,37], second-order Monte Carlo techniques, and robust Bayesian or Bayesian sensitivity analysis [38,39] which allows for the recognition of incertitude within Bayesian inference [36]. Models that more explicitly account for incertitude can sometimes yield results that seem disappointingly vague, but an honest portrayal of uncertainty is always informative.

However, catastrophe model users typically come from the world of financial modeling where data are abundant and historical prices are precisely known. It is not surprising that they prefer data analysis methods that are not optimal for addressing scenarios with sparse data and large incertitude. Adopting new methods may be a difficult transition despite their utility.

3.2. Failures of Imagination

In a workshop on the West Africa Ebola epidemic, virologist Daniel Bausch noted how prior relatively small outbreaks influenced expectations, “I think if you’d asked [Ebola virus experts] … a year and a half ago ‘Are we going to have 25,000 cases of Ebola in West Africa?,’ most of us would have not said that was a likelihood” [4]. It is common for modelers to claim that their models have been “validated” because they can replicate past data. However, this is history matching, not validation [40,41]. Acknowledging this distinction creates less surprise if we encounter extreme events not previously experienced or we discover that a model contains non-stationary processes that invalidate a model for future use. In terms of pandemic catastrophe models, diseases can behave unpredictably due to evolving pathogens or changing demographic conditions. For example, the 2017 outbreak of plague in Madagascar—normally a relatively controllable bacterial disease—was exacerbated by its spread to growing urban centers.

The user’s trust in any catastrophe model depends on assurances of validation (i.e., history matching) which may be acceptable in the case of relatively stable processes, such as earthquakes, but more tentative for pandemic catastrophe models that depend on non-stationary processes [42]. After catastrophic events, modelers have an opportunity to check their model against actual losses, so it’s not surprising that catastrophe model outputs will change after these recalibrating updates. However, it can be rather difficult to detect changes in the probabilities of extreme events caused by non-stationary processes [43].

One practical output of a catastrophe model is an exceedance probability curve, which summarizes the annual probability of exceeding particular losses—most importantly, a loss that would lead to the insolvency of an insurer. To generate these curves, catastrophe models will typically simulate many potential scenarios that include both historical data as well as scenarios that attempt to address limitations in the representativeness of the historical data set. This is especially necessary for models that include known non-stationary processes such as sea level rise, changing building technology, increased coastal development, etc. However, the Ebola epidemic serves as an example of a previously unforeseen risk that would not be captured merely by modestly expanding the range of a catastrophe model’s simulations.

Another lesson from the Ebola epidemic is that the various outbreaks within the larger epidemic had very different transmission dynamics depending on the level of cooperation and trust between the international public health workers, government officials, and affected communities. The impact of these social factors is critical to any risk assessment and difficult to characterize. Furthermore, new factors and previously unknown relationships in epidemic evolution are emerging as large epidemiological data sets are analyzed [44]. It would seem that a truly representative catastrophe model of a real-world epidemic would tend towards substantial complexity. One thoughtfully constructed pandemic catastrophe model [45–47] considers a myriad of factors including: wild and domestic viral reservoirs; the virulence and transmissibility of potential pandemic pathogens; the age and density distribution of populations; seasonal impacts; social factors such as air travel, work commutes and travel restrictions; and the availability of quality health care, antiviral medications, and vaccines. Whether these are enough factors is debatable. Estimating the insurance losses from a hurricane is fairly well-known and straightforward by comparison, yet building construction and risk mitigation details frequently ignored in standard catastrophe models can have a substantial impact on risk exposure estimates [48]. Furthermore, the impact, timing, and magnitude of each factor must be estimated along with any correlation with other factors.

In the end, we are left with the question of whether to pursue a more detailed pandemic catastrophe model that might be unusably complex or a simpler model that may be missing critical factors. There is a common, but often untested, assumption that complex problems call for complex solutions and that it is always better to use complex methods that make use of all available information [49]. However, there is a history of studies showing that very simple predictive models can often have better performance than complex models [50,51].

The question of best modeling approach has been explored in recent years in the epidemiological community in a series of infectious disease forecasting challenges [52]. For example, in 2015, the Research and Policy for Infectious Disease Dynamics (RAPIDD) Ebola forecasting challenge compared the predictive abilities of eight Ebola epidemiological models against a synthetic set of data over four scenarios at multiple points in an outbreak [53]. Models varied in complexity and ranged from 2 to 27 input parameters. The results did not point toward an “ideal” level of complexity because one of the best individual performers was a two-parameter stochastic model, while the worst performer was a two-parameter logistic growth model. The overall best performer was a Bayesian average of all the models which suggests that using multiple independent models is the current best approach. However, the last challenge scenario of an uncontrolled Ebola outbreak with noisy data was poorly predicted by all models, so even multi-model averaging has its limitations.

In a similar forecasting challenge for influenza in the U.S. over seven flu seasons, almost three-quarters of the 22 models evaluated made better seasonal predictions than the historical base-line model and the best performer was again a weighted combination of other models [54]. However, none of the models replicated the observed data more than 50% of the time and the challenge did not include an influenza pandemic—the models were optimized to predict seasonal flu.

Current epidemiological modeling for emerging infectious diseases tends toward using relatively simple models that only require the estimation of a few parameters primarily due to the scarcity of data [55,56]. More complex tools for assessing the pandemic risk of influenza [57,58] have not been widely adopted—one reason being their substantial data requirements.

Ultimately, choosing the level of model detail is a professional judgement in the art of modeling until the state of the art is improved. The current best solution to the problem of potentially non-representative data, processes or models is more attitudinal than technical. In the face of ignorance, the best approach is to use multiple competing models and constantly question their assumptions and results. In the case of epidemic modeling, real-time prediction remains a difficult challenge [59]. Furthermore, longer-term forecasting and predicting the emergence of new pathogens with pandemic potential will likely remain beyond our capabilities for the near future [60]. Given the dynamic nature of factors underlying epidemic progression, there may be limits on long-term predictability of infectious disease outbreaks and new analytical methods may be needed [61].