Type I vs. Type II Errors in Venture Capital
Type I (false positive) and Type II (false negative) errors are often discussed in the context of hypothesis testing.
The decision to invest in a startup can also be thought of as a hypothesis:
Invest: “this company will succeed and outperform the market”
Don’t Invest: “this company will fail to outperform the market”
If we invest in a company that fails, we will have falsely predicted a positive outcome–in other words, a false positive or type I error. Similarly, rejecting a company that succeeds is akin to falsely predicting a negative outcome–in other words, a false negative or type II error.
Which error type is most important to minimize in investing? Pulak Prasad provides a useful framework for answering this question with a focus on public company investing in What I Learned About Investing from Darwin.
Thought Experiment & Framework
Imagine a universe of public company stocks where 25% of the companies will outperform the market and 75% will underperform (this seems reasonable given 17% of the S&P 500's constituents have outperformed the index itself over the past year and 26% have outperformed over the last 3 months). Let’s assume we are much better than random at predicting whether a company outperforms the market. Of the companies we evaluate that go on to outperform the market, we invest in 80% of them. And, of the companies we evaluate that go on to underperform the market, we reject 80%.
This initially seems like a recipe for easy success.
Since we will invest in 80% of the companies we evaluate that go on to outperform the market, that means we will reject 20% of the companies that fall into this bucket. This is our Type II Error Rate or our False Negative Rate.
Since we reject 80% of the companies we evaluate that ultimately underperform the market, we also invest in 20% of the companies in this bucket. This is our Type I Error or False Positive Rate.
Specific Example
Assume a universe of 1,000 companies. If 25% are going to outperform the market, that means 250 companies represent “Good” investments. If our Type II Error Rate (the probability we reject a “Good” company) is 20%, then the probability we invest in a “Good” company is 80%, meaning we invest in 80%*250 = 200 companies that go on to outperform the market. We can continue this arithmetic to model out the expected success rate across our portfolio.
If looking to hone one's investment performance, is it better to minimize Type I Errors or Type II Errors?
To minimize Type I Errors would be to minimize the likelihood of investing in companies that ultimately underperform the market. Similarly, minimizing Type II Errors would minimize the likelihood of rejecting companies that ultimately outperform the market.
It turns out that reducing our Type II Error Rate to 10% only increases our overall success rate from 57% to 60%, while making the same reduction to our Type I Error Rate increases our overall success rate from 57% to 73%.
Ultimately, in an investment universe with more Bad companies than Good, minimizing the likelihood of investing in Bad companies (Type I Error) will always add more to the overall portfolio success rate than minimizing the likelihood of rejecting Good companies (Type II Error).
Venture Capital
Venture Capital investing differs from public investing in two important ways with respect to this thought experiment:
A much smaller percentage of the companies go on to succeed
Those that succeed have larger outcomes
We can imagine evaluating 1,000 companies with intentions to write $1mm checks into those we think will succeed. Companies that go on to succeed will produce 5x returns (i.e. we invest $1mm and get back $5mm) while those that fail will go on to produce 0x returns (i.e. we invest $1mm and get back $0).
We can evaluate how this strategy will unfold and how decreases in our Type I or Type II Error Rates will impact performance.
We see that reducing our Type I error rate to 10% improves our Markup On Invested Capital (MOIC) by 27%, while making a similar adjustment to our Type II error rate increases MOIC by just 5%.
But this assumes 25% of companies are Good, which may be valid for public markets but certainly not at the Seed stage.
In the world of early-stage startups, fewer than 10% go on to succeed.
With a smaller pool of Good companies within the investment universe, reducing our type I error rate now increases MOIC by 53% while the type II error reduction improves MOIC by only 8%
We can illustrate this relationship by asking ourselves, “Given a pool of $10mm to invest and the ability to reduce or Type I or Type II error rate from 20% to 10%, how much incremental investment return to we expect to generate depending on the % of companies in the universe that are “Bad”?
Conclusion
Given an investment universe where the majority of companies will underperform the market, reducing our rate of false positives (investing in bad companies) will enhance overall success more than reducing our rate of false negatives (rejecting good companies).
As the proportion of good companies decreases and bad companies increases, the value of mitigating Type I errors grows significantly greater than that of mitigating Type II errors.