Modeling R&D Risk (Part 1): Monte Carlo Simulation

Biotech Risk Models

Biotech equities are derivative securities. Their value is derived from intellectual property rights (patents). However, in contrast to other types of financial derivatives, our understanding of the risks and uncertainty involved in the underlying biotech assets remains in its infancy. The lack of understanding causes biotech equities to go through booms and busts, as investor risk appetites vary with the economic cycle. This cyclicality throws a wrench in the scientific R&D process, destroying economic value and wasting potential for increased population healthspan. Thus, advancing our ability to forecast R&D risk is not only necessary for building more efficient biotech portfolios, but is also essential for advancing our healthcare.

With this post I aim to accomplish three things:

1. Briefly introduce two R&D risk models published in the academic literature (one is a simple historical success rate model, another is simulation-based)
2. Discuss the differences between the two approaches
3. Provide benchmarks for an alternative model that estimates R&D risk in continuous time using time-to-event data. I propose this model in part 2 of the blog post

Note that here I use the words “R&D risk” to denote the probability of an event complementary to the successful completion of drug development, e.g., a clinical trial failure. For simplicity, here I assume that such failure is tantamount to the financial value of the biotech equity going to zero.

Let’s begin our discussion on risk modeling techniques by looking at the familiar R&D risk drivers.

Clinical trial success rates

Most models used in the industry still rely on point estimates of R&D success probability, derived from historical clinical trial success rates. The success probabilities are usually estimated separately for each development phase and presented in a tabular format:

The trial success rates may significantly depart from industry averages depending on the exact nature of the drug and target patient population. For example, the historical rate of a biologic drug reaching an FDA approval is nearly twice that of a small molecule drug.

Clinical trial durations

Because of the time value of money (the idea that money you have now is worth more than the identical sum in the future), the duration of drug development becomes as important to valuing biotech investments, as the success rates are.

Drug development durations vary greatly, depending on the target therapeutic indication, the existing standard of care, the chosen clinical trial endpoint, regulatory precedent, and so on. In some disease areas, like oncology and rare diseases, whole clinical trial phases can sometimes be omitted to get the life-saving treatment to the patient faster.

The precision of clinical trial planning and patient recruitment forecasting is paramount to the financial viability of drug development. Big pharma companies can sometimes lose millions of dollars per day because of minor delays in clinical trial execution. And for an underfunded biotech startup, an unexpected development delay of just a few months could mean bankruptcy.

Modeling R&D risk

Risk trajectory

When building quantitative models of drug development it is customary to combine the information about clinical trial success rates and clinical trial durations into a risk trajectory, like so:

Note that development risk tends to peak in middle clinical development (Phase 2). In his landmark book on clinical trial methodology, Steven Piantadosi argues that this makes sense from both scientific and ethical perspectives.

Scientifically, the goal of clinical trials is to confirm that the scientific findings made in molecular and animal models of the disease translate to a safe and effective treatment in human patients. Perhaps because of the medicine’s emphasis on safety that goes back to the Hippocratic principle “First do no harm”, the phase 1 clinical trials have a relatively high success rate, and most clinically tested drugs fail because they are found ineffective at combating the disease in Phase 2 trials.

Ethically, the bigger, comparative clinical trials, which expose patients to unwanted side effects and preclude them from exploring other treatment options, should only be performed when the uncertainty about the treatment’s benefit is maximal. That typically happens at a point when the treatment has already been shown to be safe in a small patient population (Phase 1) but has not been tested for efficacy in a placebo-controlled setting, i.e. it happens in Phase 2 clinical testing.

Point Estimation

Point estimation of model parameters (such as drug trial failure rate and duration) as historical data averages and fractions is easy to perform. However, its main drawback is the loss of information about the uncertainty surrounding the resulting model output. Understanding this uncertainty is crucial for managing R&D risk in a portfolio of drug development projects. Such portfolio management is precisely the task that the buy-side of the drug discovery industry (the investors and the big pharma companies) faces.

Modeling of R&D uncertainty

A probabilistic approach

In their 2012 Nature Biotechnology article, a financial engineering group from MIT presented a probabilistic model of drug development to explore the feasibility of using financial securitization to reduce aggregate R&D risk for a portfolio of drug discovery projects. In their publication, Fernandez, Stein, and Lo used Markov chain Monte Carlo (MCMC) to simulate drug development outcomes. MCMC is a statistical inference technique that relies on computer simulation to generate a large number (usually tens of thousands) of data points, which can then be used to estimate the uncertainty surrounding the model predictions.

In this particular article, modeling uncertainty allowed the authors to speculate on how much project diversification is needed to ensure a positive financial return on a securitized drug portfolio, and whether or not such development is operationally and financially viable in the real world.

Simulation modeling of drug trial outcomes

Fernandez, Stein, Lo (2012) used the following transition probability matrix in order to simulate clinical trial outcomes:

On each iteration of the simulation algorithm, a simulated drug can transition from one development stage to the next, according to the probabilities specified in the matrix rows above. For example, a drug in the preclinical development state can either stay in the preclinical state with the probability of 50%, move on to phase 1 with a probability of 34.5%, or be withdrawn (fail development) with the probability of 15.5% (see the first row of the above table). Naturally, Approved and Withdrawn are terminal drug development states, from which a transition to other states is not allowed. Fernandez et al. used six months as the step size of their simulation algorithm, i.e. the minimum development time spent in each development phase before a transition to another state could occur.

In order to reconstruct the risk trajectory shown in Figure 1, I generated five thousand drug state trajectories using the simulation mechanism described above and calculated the proportion of new drug withdrawals (terminal trial failures) on each iteration of the algorithm. In my mind, this proportion should be more or less equivalent to the point estimate of the trial failure probability presented in Figure 1. Here is its graph:

As you can see, we now get confidence intervals surrounding the risk estimate. Here’s an example of how we can further use simulation to estimate how the size of our drug portfolio impacts the uncertainty of the average outcome:

Discussion

R&D risk profile of the simulation-based model

The first thing I noted in the output of the simulation-based model (Figure 2) is the rapid de-risking of the asset in the preclinical development phase, followed by a flat risk profile through the rest of the drug development cycle. This is counter to my expectation. I would expect the R&D risk to be relatively low in Phase 1 clinical testing. As discussed above, from an ethical perspective, comparative human trials should only be employed when the uncertainty about drug efficacy is maximal. Thus, the clinical R&D risk should peak in middle clinical development, i.e. Phase 2. It would then go back down, as the larger Phase 3 study confirms Phase 2 findings. In general, the risk profile of the simulation-based model (Figure 2) should resemble the baseline risk trajectory in Figure 1. Or is it not?

Comparison of the R&D risk estimates

The two graphs cannot be directly compared because the simulation results in Figure 2 charts the rolling probability of trial failure within a six-month time window. This six-month window is the step size of the simulation algorithm for which the transition probability matrix, shown in Table 2, was computed. On the contrary, Figure 1 graphs the probability of a drug failing in a given development phase, regardless of the trial’s duration.

Nevertheless, we can easily reprogram the simulation to compute the proportion of trial failures per development phase (as opposed to per six months). In which case, the two figures can be directly compared:

Here we can see a definite, dramatic difference between the two estimated risk profiles. Even though we now see some local increase in R&D risk around Phase 2 clinical development in the simulation-based risk model, it is nowhere as pronounced as seen in the point estimates computed by Paul et al. Also, the clinical development risk estimates made by Fernandez et al. appear to be lower across the board.

Sources of variation between studies

The other obvious discrepancy between the two profiles is the lack of risk estimates for the earlier drug discovery phases in the second model. I believe this is simply due to data availability reasons. The results of molecular discovery studies often remain unpublished. Most likely, the data was not available for the trials, used by Fernandez et al. to estimate the transition probability matrix in Table 2. This brings us to the likely source of variation between the two studies: the underlying clinical trial data sets.

In their estimation, Paul et al. relied on trial data from 13 big pharmaceutical companies, who are members of the Pharmaceutical Benchmarking Forum. Whereas, Fernandez, Stein, and Lo used a database of ~700 oncology drugs that were obtained by merging oncology drug records from two drug trial databases ( Clarivate and Tufts).

Thus, we know that not only the underlying data sets differed, but one of them (used by Fernandez et al.) also focused entirely on oncology. In other words, we can expect the R&D risk profiles estimated in these studies to be different. However, to tell how much the expected R&D risk profiles differ between the oncology and non-oncology indications would require further investigation (see Part 2 of the post).

Concluding thoughts

The main advantage of estimating R&D risk using Monte Carlo simulation over point estimates of historical success probabilities is that it not only finds the expected risk but also tells you something about how good that estimate is (its uncertainty). Thus, simulation is particularly useful when investigating how diversifiable the risk is across a portfolio of drug discovery projects, given their number and hypothesized correlation of outcomes.

However, this additional functionality comes at the cost of simplicity. Note that here it took me just a few minutes to code and run the simulation in Python. But I was able to do so because the transition probability matrix was already pre-computed and provided by Fernandez et al. in their publication. Estimating this matrix from the underlying data may not be trivial and may introduce additional assumptions and sources of error into the model, which bring us to the next point.

Some of the simulation results appear counter-intuitive. As discussed above, R&D risk is expected to be non-uniformly distributed between clinical trial phases. However, the R&D risk profile outputted by the simulation looks flat throughout clinical development. The simulation study also seems to underestimate R&D risk across the board, when compared to the historical R&D success rates reported by Paul et al. It is unclear whether these discrepancies are due to differences in the underlying methodologies and/or the underlying data sets.

In the next post

In Part 2 of the post, I propose an R&D risk model based on the Survival Analysis methodology. Like the simulation model above, it provides R&D risk uncertainty estimates. However, it is simpler, as it combines various R&D stages (e.g., Phase 1, Phase 2) in a single “R&D Ongoing” state. It is also fast to implement from scratch using a dedicated Survival Analysis software package with a high-level API. Last, hopefully, it helps to answer whether the observed differences between the two studies above are primarily due to the differences in their methodologies or the underlying data.

Originally published at https://lifescifin.com.