To evaluate your process when you have few positions (e.g., 7 in a long-short fund or 20 in a venture fund), you must generate many more data points by making intermediate forecasts on milestones, not just waiting for final outcomes.
Annie explains that with only 7 or 20 investments, final outcomes are too luck-driven to evaluate your process. The solution is to make explicit probability forecasts on intermediate milestones (like whether a seed company will raise a Series B at an up round) for every opportunity that comes through. With 100 companies per year and an 18-month resolution, you rapidly accumulate enough predictions to assess your judgment, stripping luck out of the evaluation. ✦ AI generated
Annie Duke · We Study Billionaires · 2024-04-14 · original ↗
plays this moment only · 37:39 — 39:16
“Was the outcome a product of luck or skill? I think you would agree that a lot of outcomes, like you've already said, are probably a combination of both of those things. But what steps do you think we need to take to better understand if our process is generating the outcome we want rather than just plain luck?”
But how do we deal with that in a situation where your portfolio might have seven positions at a time, for example, or in venture where maybe you have 20 companies or 25 companies in a fund, right? And you're generating more outcomes than the final outcome. So it goes back to what I'm saying. So let's imagine this. Let's imagine that you're an investor investing in Series A. That's your specialty. And for every single company, every company that comes into partner meeting, You have to make a forecast of the probability that company will fund at series B. Now, you can put restrictions around it, right? Like you can say the probability that it will fund at series B with an up round. So you can do whatever you want. So we can think about what are those things that are really necessary for this to survive? Because the thing is, I can guarantee you, won't ever have a fund returner if you invest at series A that doesn't invest at series B, likely not a down round, which is a bad sign. Let's take that one data point, right? But then you can actually think about what are all the data points that I now want to be making predictions about where I'm going to start to know those things more quickly. Now, what's nice about that is it does double duty. What is it massively increases the outcomes that you have so that you can get more into the thousand coin flip situation? Because let's imagine that you have 100 companies come into partner meeting in a year And you've made that, we're just talking about the one forecast right now, you've made that one forecast for all 100 companies in the space of 18 months, you're going to know for every single one of those companies, because it's public whether they do a B, how good of a predictor am I of this particular thing that really, really matters. So now you start to be able to close these feedback loops faster because you're generating so much more data that we start to remove the luck and it becomes about sort of your predictive power.
verbatim transcript · starts at 37:39