Modeling R&D Risk (Part 2): Survival Analysis
Survival Analysis Model of R&D Risk
This is the second part of the two-part series on R&D risk modeling (see part 1). With this post, I aim to accomplish three things:
- Propose an alternative model of R&D risk based on the Survival Analysis methodology
- Outline advantages of the model over the Monte Carlo simulation method
- Discuss the role of continuous-time R&D risk models in the pricing of biotech equities
Recall our discussion on the importance of estimating the duration of drug trials in part 1 of the post. The trial duration is important because it affects the timing of future cash flows, which the financial viability of the company depends upon. Survival Analysis is a collection of statistical techniques for estimating time duration until an occurrence of an event. Here we will use it to estimate R&D risk, i.e. the rate of clinical trial failure as a function of total development time.
Data Structure
I do not have access to the drug trial databases used by Paul et al. (2010) and Fernandez, Stein, Lo (2012), discussed in the previous post. However, I will use another commercial database containing clinical trial data from over 18'000 drugs. I assume that it encompasses or heavily overlaps with data used in the above studies.
For us to estimate R&D risk using a Survival Analysis technique, our data needs to conform to the following structure:
Note: A “Censored” event denotes that the development is still ongoing for that drug
R&D States
Unlike the R&D risk models discussed in part 1 of the post, the Survival Analysis model does not make a distinction between discrete R&D stages, e.g., preclinical, phase 1 clinical, etc. A single “R&D Ongoing” (aka “Censored”) state is used instead (see Table 1). Using fewer drug development states simplifies the model and introduces fewer assumptions about the efficiency and duration of the underlying drug development process in its various stages.
The Hazard function
Using Survival Analysis, we will model the rate of R&D failure, i.e., a transition from the “R&D Ongoing” to an “R&D Failed” state, as a function of time. Because we are interested in the rate of trial failure as a function of time (as opposed to the probability of not failing drug development within a given time frame), we are going to use a Survival Analysis model that estimates the hazard function. Nelson-Aalen estimator is such a model.
The following figure was obtained by fitting the Nelson-Aalen estimator to over 5'000 oncology drugs. The Monte Carlo simulation output, discussed in part 1 of the post and based on the Fernandez et al. methodology, is overlayed in orange:
Discussion
Comparison with the Monte Carlo simulation output
The survival analysis model produces a risk profile (Figure 5) that is far more congruent with my expectations, compared to that of the Monte Carlo simulation performed as in Fernandez, Stein, Lo (2012). Unlike the Monte Carlo simulated risk profile, it is not flat throughout clinical development. Instead, the trial failure risk rises and falls with the timing of clinical trial readouts, as expected. It also produces a higher estimate of R&D risk throughout the clinical development phases, showing greater concordance with the point estimates of historical R&D success reported by Paul et al. (2010). See here for the relevant discussion in the previous post
Oncology vs. Non-oncology drugs
To facilitate comparison between the Survival Analysis and Monte Carlo Simulation models, I limited the analysis in Figure 1 to oncology drugs only (Fernande et al. used data only from oncology drugs to develop their model). Figure 2 below compares the Survival Analysis estimates of the R&D risk for the oncology and non-oncology cohorts from the 18'000 drug data set:
Interestingly the risk profiles between oncology and non-oncology drugs do not differ dramatically. However, oncology studies seem to have a higher overall success rate (lower risk), a slightly longer middle clinical development phase, and a shorter overall development duration, compared to non-oncology studies. These observations are congruent with the idiosyncrasies of oncology drug development.
Limitations of the Monte Carlo method
Since the flatness of the risk profile produced by the Monte Carlo simulation cannot be explained by the fact that it was derived using oncology-only drug data, there must be a difference in the underlying modeling methodologies.
To estimate the transition probability matrix (necessary for the Monte Carlo simulation, Fernandez et al. must have averaged the clinical trial failure rates across a rolling six-month time window, the chosen step size of their simulation algorithm (see the relevant discussion in part 1 of the post). I believe that this averaging results in the loss of information about the time distribution of risk within the R&D phases and yields the flat clinical risk profile seen in Figure 1.
It still does not explain why the Monte Carlo -estimated clinical development risk is lower than that estimated by Survival Analysis (Figure 1) or that reported by Paul et al. (2010), see Figure 3 below:
See the discussion of the differences between the Fernandez et al. and Paul et al. R&D risk estimates in the previous post.
A note on competing risks
One possible explanation is that the lower Monte Carlo risk estimate is due to a smaller sample of ~700 drugs used by Fernandez et al. vs. the ~5000 drugs used in the Survival Analysis estimation here. Perhaps the smaller selection somehow biases the risk estimate towards lower risk.
Another potential source of bias is that drug approval and drug trial failure are potentially competing events. Indeed, it is unlikely that a drug that has failed a clinical trial will be approved (and vice versa). Some Survival Analysis methods (including the Nelson-Aalen estimator used here) may yield biased results when estimating event-specific probabilities in the presence of competing risks.
A good follow-up study would be to compare the Nelson-Aalen estimate of the R&D risk with that of another Survival analysis estimator that is known to produce robust estimates in a competing risk framework.
In the next post
Successful drug R&D is followed by regulatory approval and commercial drug launch. However, unlike studies of R&D risk, very few (if any) studies of drug commercialization uncertainty have been published. In the next post, I will discuss the reasons for that neglect and address the common misconceptions about the commercial side of drug development by analyzing historical sales data from over 1600 branded pharmaceuticals.
Originally published at https://lifescifin.com.
