To find your edge, you must start with the base rate (what usually happens in similar situations) before layering in your own personal experience — not the other way around.

(00:00:01) You're listening to TIP. (00:00:02) Annie Duke has one of the best minds on decision making. (00:00:05) I've talked to you on We Study Billionaires. (00:00:08) Decision making might not sound thrilling at first, but hold on tight because it's the very essence of successful investing. (00:00:13) Think about it. (00:00:14) Every trade, every hold, every decision is a product of our mental gymnastics aimed at maximizing returns and optimizing our process. (00:00:22) And Annie, with her background as a legendary poker player turned author, brings unparalleled insights into this complex arena. (00:00:28) In this episode, we're not just scratching the surface we're delving into Annie's treasure trove of wisdom, honed over years of intense study and practical experience. (00:00:36) Her books, Thinking Invests and Quit, are more than just reads they're guides to mastering decision-making in both investing and life. (00:00:44) Today we'll unpack some of our most valuable advice to investors to bring more clarity to your thinking. (00:00:48) We'll focus on our pre-commitment devices, invaluable mental tools that can save us from costly mistakes and lead us towards smarter, more profitable choices. (00:00:56) One of my biggest takeaways from our chat was how we can create data points on long-term investments. (00:01:00) This allows us to zoom into our process to make sure we are on track for a process that may be many years into the future. (00:01:07) Imagine being able to anticipate the fundamental downturns of a business in your portfolio before the market recognizes it. (00:01:13) Or think about having the ability to more easily walk away from a losing investment that is burning a hole in your pocket and eating away at your returns. (00:01:20) Annie's insights here alone are worth paying very close attention to. (00:01:24) So if you relish in the intellectual challenge of investing, this episode is tailor-made for you. (00:01:28) After listening to this episode, you'll have a number of practical tools that you can use instantly to supercharge your decision-making, not just in investing, but in every aspect of your life. (00:01:37) Now let's get right into this week's episode with Annie Duke. (00:01:43) Celebrating 10 years and more than 150 million downloads. (00:01:47) You are listening to the Investors Podcast Network. (00:01:50) Since 2014, we studied the financial markets and read the books that influence self-made billionaires the most. (00:01:57) We keep you informed and prepared for the unexpected. (00:02:00) Now for your host, Kyle Green. (00:02:12) Welcome to the Investors Podcast. (00:02:13) I'm your host, Kyle Grieve, and today we bring Annie Duke onto the show. (00:02:17) Annie, welcome to the podcast. (00:02:19) Thank you for having me. (00:02:21) So Annie has written several books on how to think more efficiently and has tied many of her awesome experiences from being a professional poker player into the books. (00:02:29) Two of her books that really stood out to me were Thinking in Bets and Quit. (00:02:32) So I'm very excited to learn more about some of the key concepts from these books and discuss how listeners of the show can apply these concepts to make better decisions. (00:02:40) So to kick things off, Annie, I'd love for you to just give your backstory in academics and poker and how it relates to your own story of quitting. (00:02:47) I started off my adult life at the University of Pennsylvania. (00:02:50) I was getting my PhD in cognitive science, which is just broadly, how do we as humans create models of the world? (00:02:58) How are we sort of interacting with the world and (00:03:02) learning, judgment and decision making, those kinds of things would go under that. (00:03:07) I'd done my major area exams, I was out on the job market, I had my thesis research finished, and I got sick. (00:03:15) So I had sort of something that was kind of chronic and it turned acute. (00:03:20) And I actually ended up in the hospital for a couple of weeks with it. (00:03:24) And it was just very clear, like I needed to take a little bit of time off. (00:03:28) And so I had to cancel my job talks. (00:03:30) And (00:03:31) fully intending to become an academic and get a tenure track position in academics as one does. (00:03:37) So I took off and then I was going to come back the next year and finish up the next year and go back out on the market. (00:03:42) It was during that time when I was taking time off after being there for five years that I started playing poker. (00:03:49) And I knew a little bit about poker before that because my brother was already a professional poker player and I've watched him play quite a bit. (00:03:57) He had brought me out to Las Vegas a few times on just vacations during graduate school, vacations I could not afford on a fellowship. (00:04:04) And so I played like a little bit, but not seriously. (00:04:08) And he suggested that maybe I do that because I just honestly, like I really needed money. (00:04:13) I didn't have my fellowship anymore. (00:04:15) I didn't come from a family that had money. (00:04:17) So I just needed to be able to support myself. (00:04:19) So I started playing poker, which was really kind of ideal for me because I didn't know how I was going to feel from day-to-day. (00:04:25) I was still recovering from this illness. (00:04:27) And because I was going to go back to academics, I didn't want to start a career or something like that. (00:04:32) So poker seemed like a good thing to do. (00:04:34) In the meantime, my brother gave me some tips. (00:04:37) I already knew some about it from watching him and talking to him about it. (00:04:40) And when I started playing, I just started doing really well really quickly. (00:04:44) I was up actually in Montana, a very weird place to be playing poker, but they had legalized poker. (00:04:50) And so I would go to downtown Billings and go play poker, which was kind of strange. (00:04:58) Even stranger when you realize that at the time that I did this, poker was not on television. (00:05:04) There was no internet poker. (00:05:05) So I think it's hard for people to (00:05:08) kind of understand what it was like in the olden days in the sense, but pre-2002, that poker players weren't cool, they weren't on television, people didn't know who they were. (00:05:19) They didn't even understand the most basic thing, which was that you could make your money playing poker. (00:05:24) When I told people that I was playing poker, it would generally end up somewhere in the, oh, your husband must make a lot of money, or are you going to Gamblers Anonymous? (00:05:33) You know, whereas like if you flash forward a decade from there, people were like, oh my God, that's so cool. (00:05:38) So it was kind of a strange thing to do, but I loved it. (00:05:42) I just, I loved it. (00:05:44) seemed to me to be just this very practical, real-time, high-stakes application of the things that I was learning in psychology, which is how do you actually make good decisions in a kind of environment where there's so much uncertainty? (00:06:00) And that's actually a lot of what I was studying in graduate school. (00:06:04) And the uncertainty in poker is coming from two places. (00:06:06) One is there's just a lot of luck (00:06:08) which obviously people who are in markets know, and poker is just a market. (00:06:12) And then there's also a lot of hidden information, which should also sound very familiar. (00:06:16) So it's just a very high vol and relatively low information setting. (00:06:20) So it's those types of environments where our decision making can go really bad in the form of predictable human error that might occur. (00:06:29) So for anybody who's familiar with, for example, Daniel Kahneman, (00:06:32) That's really what his life's work is about, things like confirmation bias and overconfidence. (00:06:36) And in those types of environments, that's where those things get the worst. (00:06:40) So I was really interested in just really learning the game and trying to solve for these issues. (00:06:46) And for eight years, I didn't go back to graduate school. (00:06:49) I was as ABD as you could get. (00:06:51) My dissertation work actually even got published. (00:06:53) But for eight years, I just dove headlong into poker. (00:06:57) Then in 2002, (00:07:00) When poker sort of got on television, I got asked by hedge fund to come speak to their options traders about how poker might inform their thinking about risk. (00:07:09) And I took it in a little bit of a different direction because I'm kind of a cognitive scientist at heart. (00:07:14) And I talked about how. (00:07:17) The track that you're on, whether you've been winning or losing recently, really distorts your risk attitudes on your next decision, which is a really big problem that poker players have as well. (00:07:25) So I really kind of talked about that issue. (00:07:28) Broadly in poker, we might put that under a category called tilt, which is like when your brain sort of stops working because you're emotional about things that have happened in the past. (00:07:38) So I gave that talk. (00:07:39) It was really fun. (00:07:40) And that person, the founder, the managing director of that hedge fund, ended up (00:07:46) recommending, starting to recommend me to other people. (00:07:50) And then obviously as poker got on television and I was kind of one of the OGs, I started getting asked to do like events for businesses. (00:08:00) And mostly it started off with people just wanting me to come in, like play poker for their like retreats. (00:08:06) And I started really pushing my manager to offer talks because the thing that really happened to me when I gave that first talk was I remembered some, I remember 2 things. (00:08:16) The first thing is I remembered that I really do love cognitive science. (00:08:20) But the more important thing that I remembered was I love teaching. (00:08:23) And I hadn't been doing it, obviously. (00:08:24) I'd been teaching myself, but I hadn't been teaching other people, and I really wanted to do more of that. (00:08:29) So I started sort of getting asked to come do these talks, originally a little bit me pushing it, but then just through word of mouth and whatnot, started building up that business. (00:08:39) Actually ended up teaching a little poker on the side too, because I found a way to do a lot of teaching. (00:08:44) And then sort of around 2012, so I'm doing that in parallel, right? (00:08:47) I'm a poker player. (00:08:49) And then I also have developed this whole other thing that I'm doing, which is thinking about the intersection between cognitive science and poker and how those two disciplines might inform each other to make us actually better at both of them. (00:09:02) So I'm now doing those things in parallel. (00:09:06) I start getting asked by some of the people that I speak to if I do consulting, that might be interesting. (00:09:12) So I start doing some of that. (00:09:14) And then in 2012, really made the decision to retire from poker completely so that I could focus on this other thing that I was doing and that I did. (00:09:25) And so my life now is I have a very small roster of clients. (00:09:31) I keep them small because I embed. (00:09:34) I don't do short term projects with people. (00:09:37) the client that I've been with the shortest amount of time, except I just took on a brand new one, so I'll exclude them. (00:09:43) But in terms of any of my older clients, the shortest amount of time that I've been with any of them is now three years. (00:09:49) So I really go deep and long with the people that I work with. (00:09:53) And then I also just was really kind of burning to start writing down the things that I was talking about, what I was doing in terms of the work in my consulting work. (00:10:05) And that became first thinking in bets, then how to decide, and then quit. (00:10:10) Thinking about my next book right now, I'm just starting research on it actually. (00:10:13) And then the other fun thing that I did was last year, I had been doing research with Phil Tetlock and Barb Mellers, and he wrote Super Forecasting, which I'm sure you're familiar with. (00:10:24) Both of them completely brilliant. (00:10:26) And during COVID, they had asked me to collaborate with them on some research. (00:10:31) on forecasting. (00:10:32) And so I did that and I sort of became lead investigator on series of four pretty large scale studies. (00:10:39) We found really fun results. (00:10:40) And at the end of it, Phil said to me, why don't you just write this up? (00:10:45) Like it's more than most people do for a dissertation by a lot. (00:10:49) And so last year on June 15th, (00:10:51) I successfully defended my dissertation and was officially PhD on August 4th. (00:10:57) So you mentioned Eric Seidel in Thinking in Betts and some of the key lessons he imparted to you. (00:11:02) And you said that you had a crush on his intellect at the time. (00:11:05) So I really enjoy learning about mentors of people I highly respect like yourself. (00:11:11) So I'm just interested, I mean, I'm probably sure you could go hours speaking about the mentors, but maybe could you give me a little bit of information about some of the mentors who had the biggest impacts on you, whether that's inside the realm of poker or outside and why they were so influential for you? (00:11:26) academically, it has to start with Barbara Landau, who is now at Johns Hopkins. (00:11:31) She was at Columbia at the time. (00:11:33) It was her first job out of graduate school, I believe, actually, in a tenured track position. (00:11:37) And (00:11:38) The first week of college, I was looking for a work-study job, and she was looking for a research assistant, and the rest is history. (00:11:47) And so I started working with her, and I actually stayed with her for the whole four years that I was in college. (00:11:52) And she had actually just come from Penn, where her advisors were Lila and Henry Gleitman. (00:11:58) And she was amazing because she did something that a really good mentor will do, which is I really wanted to stay at Columbia, I did not want to leave New York, (00:12:08) So I wanted to do graduate school and do my dissertation under her. (00:12:13) And she sort of put her foot down and said, no, you really have to go to Penn. (00:12:17) I don't think it's good for you to stay with the same person for eight years, which I thought was, or nine years actually, which I thought was pretty amazing. (00:12:25) I did end up going to Penn and I studied with Lila and Henry. (00:12:27) So they're my next really big mentors in my life. (00:12:31) And just, you know, Lila, I was close with until she was 91. (00:12:36) just really showed me a model of not just incredible intellect, but also compassion. (00:12:44) And again, as a good mentor does, while I was, I sort of felt like I had really let her and Henry down by leaving graduate school and going off to become a poker player. (00:12:55) She was just so beaming with pride about it. (00:12:58) So for her, successful mentorship was me going off and (00:13:02) excelling at whatever it was that I did. (00:13:04) And it was funny because I had such a, I was so focused on the letting down because I didn't follow in her footsteps that I didn't think like, well, of course you would be proud of a student who went off and became a world champion at the thing that they were doing. (00:13:15) So I have to, obviously, there are huge influences in my life. (00:13:19) In poker, my brother, clearly he's the one who taught me to play. (00:13:22) He was the person who I most kind of like learned from and bounced hands off of. (00:13:27) And then Eric Seidel would be the second one who (00:13:30) taught me a lot about how to play poker, but more his mentorship was so much in that kind of like emotional control and how are you processing like good and bad outcomes and how are you treating like your fellow human beings in terms of the way that you're communicating to them and sort of generosity of intellect and that kind of stuff. (00:13:50) Like he was more, my brother was more teaching me poker. (00:13:53) And Eric Seidel was teaching me much more about how to behave as a poker player, which I think it was so important because you do do so much, there is so much emotion in poker that if you can't get that in check, you're going to be in really big trouble. (00:14:06) And he's so good at it, you know. (00:14:09) So he's such a huge influence for me. (00:14:11) And then we start to get to, as I move into this next, the phase that I'm in right now, which, you know, in terms of writing and, you know, and the consulting work and so on and so forth, I mean, obviously Phil and Barb, (00:14:23) Phil Tetlockenbard Mellers in terms of the mentorship around like my dissertation and actually completing that work. (00:14:29) But then the people that you mentioned, you know, Danny Kahneman has been an amazing mentor to me. (00:14:34) Michael Mobison and I are just, you know, catch up all the time. (00:14:38) He's one of my idols. (00:14:40) I, you know, he, I think that just the way that he thinks and his intellectual curiosity is something to completely aspire to. (00:14:48) And so he's definitely been a mentor. (00:14:51) to me in that way as well. (00:14:54) and of course at the age where mentorship and friendship a little bit gets melded together, do you know what I mean? (00:15:00) Because they're also like, Michael's also a very good friend and Danny's a friend. (00:15:04) So, and then Katie Milkman sort of in that category where like, I look to her for mentorship, but she's also, you know, my friend. (00:15:11) So I'm kind of at that age where it sort of mixes together a little bit. (00:15:15) And then I would say the last person in terms of mentorship would be my husband. (00:15:20) who really helps to guide me. (00:15:21) And a lot of what he does is help me to find permission to say no to things, which I really do need a lot of mentorship on. (00:15:28) So I've been very lucky actually, and I know I'm leaving people out, but I've been very lucky to have some just really awesome, like really incredible mentors in my life. (00:15:38) And it's so important, obviously, to where I am now. (00:15:42) Well, that's an incredible list. (00:15:44) In your chat with my colleague William Green, you mentioned that we all think probabilistically, even if it's not explicit. (00:15:50) So you used a great example of when we drive to work and decide which route to take. (00:15:55) We're using probabilistic reasoning to determine which route is going to get us there the quickest. (00:16:00) So we are using a form of, an implicit form, sorry, of expected value to help us determine which route makes the most sense, given the information that we know at the time. (00:16:10) But as you pointed out, the real magic happens. (00:16:12) when we explicitly use probabilistic thinking. (00:16:15) So I'm interested from an investing lens, what can investors take from this lesson about probabilistic thinking to help them minimize risk in investing? (00:16:25) First of all, let me just say this, that if you have the expected value right, risk becomes a much less important issue. (00:16:34) This is a concept that really was hit home for me from Jeff Yas. (00:16:39) who's the founder of Susquehanna International Group. (00:16:42) And he actually said, Rich Smith, like what I care about is I'm worried that I think I'm winning when I'm actually losing. (00:16:49) There's actually Don Moore and Max Bazerman have written about this where they think everybody should be much more focused on expected value, right? (00:16:56) Because if the expectancy is positive, you're probably not going to make too big a mistake. (00:17:02) Now, obviously, you have to think about risk in the sense of, am I going to have more money to churn through (00:17:09) this positive expectancy thing. (00:17:12) We definitely want to think about it when we're in that sort of risk of ruin category. (00:17:17) So that's going to be particularly important when we're in a higher volatility situation. (00:17:22) So in poker, we do actually think about this quite a bit. (00:17:25) But you have to know what your edge is. (00:17:28) I mean, that's the thing, right? (00:17:30) So you have to have a good sense of what your edge is, which is really an expected value problem, right? (00:17:36) So, because you can't calculate expected value without knowing what your edge is, right? (00:17:40) So you have to know what your edge is in order to start to manage risk well. (00:17:44) Obviously, assuming that you're in a low ball, if you're in a low ball situation, you're probably never going to make too much of a mistake as long as you have a positive expectancy. (00:17:53) But if you're in a high ball situation, you do actually have to start to think about that more deeply. (00:17:57) In poker, we were just applying something that was very similar to Kelly, basically, which is like essentially bet your edge. (00:18:05) And a little bit, the way that I think about Kelly is, first of all, just how confident are you of what your edge is? (00:18:11) Because I think that the less confident that you are about your edge, the more that you should be moving into like half Kelly, quarter Kelly, that kind of thing. (00:18:20) Because you really just have to give yourself a cushion on that. (00:18:22) And then there's just also your tolerance for going broke. (00:18:26) How easy is the money to replace and things like that? (00:18:28) So I tended, in poker, I tended to bet somewhere around (00:18:34) never more than 5% of my total bankroll would be in play at any time. (00:18:39) But usually it was more in the 2.5% range. (00:18:42) That would have been a foolish to half Kelly kind of situation. (00:18:47) But again, the thing about all of this is that with risk, it's like you can't manage risk unless you know what your edge is. (00:18:53) You have to know what your expected value is in order to be able to manage risk. (00:18:57) And I think the big mistake that people make in investing is that you have risk managers and there are formulas that you can apply to risk. (00:19:04) And I think they make an assumption about their expected value and they get really, really focused on the risk management side of things because it's a problem that I think is easier to solve. (00:19:16) In other words, if you have an assumption about what your EV is and what the vol is, right, now you can actually just apply a formula and you can start to get into some (00:19:25) sort of dive headlong into the risk management side of things. (00:19:29) And that's Jeff Yass's point, is that he feels like people are making too many assumptions about the EV part. (00:19:35) Just like what is your edge in the 1st place, right? (00:19:38) And I think everybody comes in assuming they have an edge. (00:19:42) And then they go to all of these risk management formulas, right? (00:19:46) And he's like, I don't even care about that. (00:19:47) Like, I'm just so afraid. (00:19:50) that I think I'm winning when I'm actually losing. (00:19:52) And I think that's a thing that people really need to be thinking about when it comes to expected value. (00:19:56) And it's one of the reasons why you really want to make it explicit. (00:19:59) So how do you. (00:20:00) How would someone really go over finding their edge? (00:20:03) I mean, I guess you'd have to just, because it's all from experience, right? (00:20:07) And you can't just kind of snap your fingers and have this number in your head. (00:20:11) You kind of have to base it off of your prior experiences and kind of go from there. (00:20:15) Is that how you would coach someone to do it? (00:20:18) So prior experience matters, but not as much as the base rate. (00:20:22) This is something Michael Mobison just like hits home so well. (00:20:26) The thing that he is always saying, and I've seen a lot of talks that he's given, (00:20:30) is the way we normally go about making decisions is we think about the problem that we're approaching and then our own experiences. (00:20:38) And then we use that to decide what we think our EV is going to be. (00:20:41) So if I'm trading and I'm thinking about a particular thesis that I might trade or whatever, I'm going to be thinking about how smart I am and how unique the thesis is and so and so forth. (00:20:53) And my past experiences with winning and (00:20:56) all of that. (00:20:57) But what I should really do to start with is his point is before I get to my own experience, I have to start with base rate. (00:21:03) Let me give you an example. (00:21:05) Let's imagine that I'm thinking about opening up a restaurant. (00:21:09) And I've worked in restaurants before, and the chefs that I've worked for have said that I'm amazing and I'm a great cook. (00:21:17) and the customers are always complimenting my food. (00:21:21) And now I'm opening up my own joint and I'm thinking about how much people love my food and how busy the other restaurants I've been in have been. (00:21:32) And I estimate that the probability that my restaurant is going to thrive is 80%. (00:21:40) Because notice I'm thinking about all these things that are personal to my experience and how much people love the food that I make and how good a cook I am and how well the restaurants that I've been in have done and so on and so forth that I've cooked in have done. (00:21:53) But that's not the place that I want to start. (00:21:55) That stuff matters. (00:21:57) But what I actually want to start is with the base rate. (00:21:59) So what I want to think about is what usually happens in a situation that's similar to the one that I'm considering. (00:22:06) So this is going to give you, in investor speak, beta. (00:22:10) You need to know what beta is. (00:22:12) In restaurants, if I think about, if I actually look up what percentage of first-time restaurants are still open at the end of the first year, survive a year, it's 40%. (00:22:22) So now I'm going to start with 40%. (00:22:24) And now I can say all those same things. (00:22:28) I'm a great cook. (00:22:29) People love my food. (00:22:30) Customers are always complimenting me. (00:22:33) at the restaurants that I've cooked for, the chefs that I've worked with have said that I'm amazing and I should open my own joint. (00:22:40) And so I think that my chances are better than 40%, but not 80%, right? (00:22:45) Like I'm not doubling my chances here. (00:22:48) So this allows me to get grounded in a reality that I can now toggle up or down from. (00:22:57) So we can look at that, we can find the base rate and then we can say, (00:23:02) Do my particular circumstances, do I think it makes it more likely or less likely that I'm going to succeed independent of the base rate? (00:23:11) Now, we can use this concept in terms of historical averages also. (00:23:16) So if I want to make a guess what my Q2 target should be, I shouldn't go by what my board wants it to be. (00:23:26) I should look at what has on average growth been (00:23:30) year over year. (00:23:32) And then I can actually historically look, because again, base rates are interesting, right? (00:23:36) There's some art to base rates because you have to find the right, what's called reference class. (00:23:41) You have to find the right sort of situation similar to the one that you're considering. (00:23:45) So I can look at growth year over year, but then I can also look at what has historically happened between Q1 and Q2. (00:23:52) If my company has been around long enough, I can look what's historically happened within my own company because there may be seasonality, for example, or I can look (00:24:00) industry in general. (00:24:02) So this is now you can see it's going to help me get a starting point. (00:24:06) So let's imagine that quarter over quarter growth in net new ARR has been recently 10%. (00:24:15) But then I also see that there's a seasonality component where my sales actually go up in June, Q2 more than I would expect them to go up between Q3 and Q4 and Q1. (00:24:28) So maybe I bring it up then and I say, actually the seasonality is working in my favor because that usually gives you an extra 50%. (00:24:33) So I say, okay, I think it's actually going to go up 15%. (00:24:37) But my sales leader is about to exit. (00:24:39) So that's something else that's personal to me. (00:24:42) My sales leader is about to exit, so maybe I should toggle it back down. (00:24:45) So you can see how you can now use this to ground you in reality to get to a more reasonable place in terms of what your forecast might be. (00:24:55) Let's take a quick break and hear from today's sponsors. (00:24:58) Curious about online trading but haven't taken the first step yet? (00:25:01) You're not alone. (00:25:02) And Plus 500 Futures is a great place to start. (00:25:06) The futures markets are moving fast and with Plus 500 you can explore popular assets like oil, gold, S&P 500, Bitcoin, and more. (00:25:14) From crypto to commodities, there's always something happening. (00:25:18) The platform is super easy to use so you can trade on the go right from your phone. (00:25:23) You can get started with just $100 and jump into the action. (00:25:26) See something interesting? (00:25:27) Once your account is open, you can trade it in just a couple of clicks. (00:25:31) And if you're not quite ready yet, you can practice with a free demo account. (00:25:34) No risk, no pressure. (00:25:36) With 20 years of experience, Plus500 makes trading more accessible than ever. (00:25:41) Check it out at plus500.com. (00:25:44) Trading in futures involves risks of loss and is not suitable for everyone. (00:25:48) Not all applicants will qualify. (00:25:50) Plus500. (00:25:51) It's trading with a plus. (00:25:53) They say every day your business is late to AI, you fall two days behind and the competition, they're only moving faster. (00:26:01) Fortunately, there's NetSuite Next. (00:26:03) You probably already know NetSuite, the AI-powered business suite that securely connects all your data, financials, inventory, commerce, HR, and CRM all in one source of truth, trusted by over 43,000 customers. (00:26:17) NetSuite Next is the next huge leap because now AI is built to do everything you do. (00:26:23) It surfaces custom insights throughout your day. (00:26:26) AI agents work alongside you on the routine stuff. (00:26:29) And anytime you've got a question about anything, you just ask like you're talking to a colleague. (00:26:33) And I use it and really I think you should too. (00:26:36) For the first time ever, you can try NetSuite Next for free. (00:26:39) If your revenue is at least in the seven figures, go to netsuite.ai/tip. (00:26:46) Built for every industry. (00:26:47) For every boardroom, netsuite.ai TIP. (00:26:52) As investors, we spend all day studying great businesses, but some of the best returns come from building one yourself. (00:26:59) If you've been sitting on a business idea, Shopify makes it easy to actually make it happen. (00:27:07) Everything you need to start selling is included and ready from day one. (00:27:10) Take the moment your first customer is ready to pay. (00:27:13) Shopify checkout helps more of them finish the purchase. (00:27:16) And when they come back, their details are already saved. (00:27:19) One tap and they're done. (00:27:22) And because Shopify handles the setup and the checkout. (00:27:24) you get more time to focus on what actually grows the business, plus the tools to do it. (00:27:30) If I were starting a business, I would absolutely use Shopify myself. (00:27:35) It powers millions of businesses worldwide, from names like Mattel and Gymshark to founders just getting started at shopify.com slash TIP. (00:27:47) With Shopify, nothing stands between your idea and a real business, so go make it one. (00:27:53) Start your free trial at shopify.com/tip. (00:27:58) Start your free trial at shopify.com/tip. (00:28:04) So I know that feedback loops are your obsession. (00:28:07) So when I think about feedback loops myself, I think a lot about jujitsu, an activity that I'm very fond of. (00:28:12) So the beautiful part about jujitsu is that feedback loops are pretty much instantaneous. (00:28:17) I can try a new move or a new submission and instantly I know it works, my opponent taps or it doesn't work, they escape. (00:28:25) But obviously in investing, those feedback loops aren't instantaneous. (00:28:30) You might make a decision today and you might not actually know if that decision was good or bad a couple years out. (00:28:38) So I'm really interested in knowing how do you best close feedback loops on decisions where the outcomes won't be clear for a few years? (00:28:47) Oh, I'm so excited that you asked me that because it's one of my favorite things to talk about. (00:28:50) Two of my very long-term clients are (00:28:54) venture firms. (00:28:55) And they're both early stage. (00:28:57) One is focused on seed, that's first round capital partners, I'm a special partner there. (00:29:02) And then the other is focused more kind of in the series B area, and that would be Renegade. (00:29:09) Love them both. (00:29:10) Prior to my working with them, I was invited to talk to partners at a variety of different venture firms. (00:29:18) And they all kind of said two things to me, which I thought was interesting, because I heard it echo through the whole industry. (00:29:25) One is, well, you can't really close feedback loops appropriately in the way that you talk about in Thinking and Bets. (00:29:32) So this was after Thinking and Bets came out, where I talk, you know, there's this obsession in Thinking and Bets about closing feedback loops. (00:29:38) So you can't do that when there's power law, like when power law applies. (00:29:42) And just for those people who might not know what power law is, it's when you have a very small number of winners that win a ton, but most things die. (00:29:52) So this should sound very much like Venture. (00:29:56) It's actually a little bit like social media where like 2% of the users are producing all of the content and everybody else is kind of quiet. (00:30:03) The power law applies in a variety of different places, but it definitely applies to venture. (00:30:07) Their point was, well, if everything's dying and you only have a couple winners, you could never tell anything about the quality of your decisions. (00:30:16) The second thing that they said was the feedback loops are too long. (00:30:19) If you're investing at seed, it's going to be five or 10 years, really, before you get whatever the outcome is. (00:30:26) I said the same thing to all of them, and it was only when I got to first round. (00:30:31) and to Renegade that they went, Oh, okay, I hear what you're saying. (00:30:38) This is why I work with them. (00:30:40) And in particular, Renegade was new. (00:30:42) Josh Koppelman was the one that I originally talked to who's the founder of FirstRound and he's so tremendously successful. (00:30:48) So I just want to give like a big shout out to him because somebody that successful doesn't need to be open to changing the way that they think about things, right? (00:30:56) And very often aren't open and he was completely open (00:30:59) to changing the way that he was thinking about this. (00:31:01) So let me tell you what I said to them, because this is the answer to your questions. (00:31:05) I said, What do you mean the feedback loops are long? (00:31:08) And they said, What do you mean? (00:31:09) We don't know if it excerpts for... (00:31:11) And I said, I'm sorry, do you invest in the company and then you go to sleep like Rip Van Winkle and 10 years later you wake up and you find out what happened? (00:31:21) When you invest in a company, a couple of things are true. (00:31:25) Two different categories of things are true. (00:31:27) Both of which allow you to close the feedback loop more quickly. (00:31:31) Thing number one is that you actually know objective things about the company. (00:31:36) You know whether ARR is growing. (00:31:40) You know whether they're hiring top talent and retaining the top talent. (00:31:44) You know whether they fund a series A. (00:31:47) You know whether it's an up round, a flat round, a down round. (00:31:50) You know what the quality of the syndicate is, same thing for B, same thing for C. (00:31:55) If you're, depending on the speed of the market, if you invest at seed, for example, you're going to know something very significant about that company between six and 16 months later. (00:32:06) That sounds like a lot faster than 10 years, the first thing that you know. (00:32:10) The second thing, and this is true across all investing, is that you're investing in the company because you're making a particular bet, and the bet is your thesis. (00:32:20) If it's in the market, you're saying, I think that I know something that the market doesn't know. (00:32:26) Why do I know that that's what your thesis is? (00:32:29) Because otherwise you would be indexing the market. (00:32:31) You're not indexing the market. (00:32:33) So you're saying, I believe that the market has this mispriced temporarily. (00:32:38) I believe the market is efficient, but not every single moment. (00:32:42) It's overall efficient. (00:32:44) So I believe that at this moment, (00:32:47) When it comes to this stock or whatever, this stock or this option, the market does not have this price efficiently. (00:32:54) So you have a thesis about why that is true. (00:32:57) It's true when you're investing in a company. (00:32:59) I believe that this market is going to be a great market to be in. (00:33:04) This product is going to have a competitive advantage. (00:33:07) They're going to execute in this particular way, so on, so forth. (00:33:11) And those kinds of things, you can find out very quickly, even when you're investing in a seed stage company. (00:33:18) You can see like, are they executing in the way that I thought they were? (00:33:21) Is their product gaining traction? (00:33:23) Like, so on, so forth, right? (00:33:25) So all of these things like, look, is it like poker or jujitsu where you're going to find out two seconds later? (00:33:33) No, but you're going to find out way more than 10 years. (00:33:37) That's what we're obsessed with, is how are we thinking about the way that we can grade these companies as they develop, where we know things about the quality of the decision long before 10 years is up. (00:33:57) The other thing that's really important to know is that, again, because of the power law issue, (00:34:03) There's lots and lots of companies that die that were great investments. (00:34:08) There's a lot of luck that's happening. (00:34:10) Do you have a company that COVID, it just destroys or whatever? (00:34:17) You want to be able to see the companies that you wanted to have in your portfolio, regardless of whether they ended up being fund returners or exited for over a billion dollars. (00:34:26) You can only do that if you're actually tracking these things that you know are necessary but not sufficient. (00:34:33) for them to get a billion dollars to a billion dollars. (00:34:36) And that all translates perfectly onto, for example, people who have long-short funds, right, where most of the long, if you're a value investor, is long hold. (00:34:49) And so you need to start saying, okay, but (00:34:52) I don't want to not know for five years what are the things that's happening with this company that I've invested in, given what my thesis was for why the market was inefficient here, and are those things unfolding? (00:35:05) So I just never accept, ever, that the feedback loop is too long to be able to do anything with. (00:35:12) I'm sorry, I got really passionate about that, but it's like that's my thing. (00:35:17) That's the thing that frustrates me the most. (00:35:19) So I apologize for getting so excited. (00:35:23) No, please don't. (00:35:24) That was an excellent explanation. (00:35:26) In your conversation with Howard Marks, you discussed a lot about the role of luck that you just brought up and how that plays in outcomes. (00:35:32) So this has been a fascinating area for me that I spend a lot of time thinking about. (00:35:35) So you said, quote, in the short run, there's just way too much luck. (00:35:39) So as an example, if Howard and I were betting and he said, I'm gonna lay you 2 to 1 on a coin flip, a coin being 50-50, I'm gonna make 50% on every dollar that I bet there. (00:35:49) If I call heads and it lands tails, it means nothing. (00:35:52) Now, if we were to do it a thousand times and I kept losing, then I could start to draw some conclusions from the outcome, like maybe Howard isn't using a fair coin. (00:36:00) From an investing view, this shows how hard it can be to evaluate ourselves on a single decision. (00:36:06) Was the outcome a product of luck or skill? (00:36:09) I think you would agree that a lot of outcomes, like you've already said, are probably a combination of both of those things. (00:36:14) But what steps do you think we need to take to better understand if our process is generating the outcome we want rather than just plain luck? (00:36:22) Obviously, like in poker at the end of a year, assuming you've played 1,000 to 1,500 hours of poker, your results are going to have very little influence of luck on them. (00:36:33) Very different than one hand of poker, right? (00:36:35) What we're trying to do is generate enough data points to start understanding the luck skill differential. (00:36:42) This goes back to what I just said. (00:36:44) So let's imagine that I'm a venture fund, and I'm only going to make 20 bets in a fund, or I'm a long-short investor, and maybe at any given time, I'm going to have seven positions. (00:36:56) So now you're sitting here going, well, how am I ever supposed to know if my process is good? (00:37:02) seven coin flips also doesn't tell you very much. (00:37:06) So if I'm just waiting for the outcome of those things, it's not going to help me very much. (00:37:11) So a great investor, if their fund has 20 companies in it, could have a fund that returns 0.8, could have a fund that returns 40x, and it's the exact same investor. (00:37:23) So it's like, did you get Uber in there? (00:37:27) So the way that we deal with that is to create more outcomes. (00:37:32) in those types of environments. (00:37:33) So notice I don't have to do that in poker because in poker, I'm churning money through that system so quickly that I'm getting enough outcomes to start to draw conclusions, particularly if I combine that with really deep dives on like a process or my thinking process during a hand where I'm hiding the outcome from the person that I'm doing that deep dive with. (00:37:54) So I can kind of do that in poker, right? (00:37:57) But how do we deal with that in (00:37:59) a situation where your portfolio might have seven positions at a time, for example, or in venture where maybe you have 20 companies or 25 companies in a fund, right? (00:38:09) And you're generating more outcomes than the final outcome. (00:38:14) So it goes back to what I'm saying. (00:38:15) So let's imagine this. (00:38:17) Let's imagine that you're an investor investing in Series A. (00:38:21) That's your specialty. (00:38:23) And for every single company, every company that comes into partner meeting, (00:38:29) You have to make a forecast of the probability that company will fund at series B. (00:38:35) Now, you can put restrictions around it, right? (00:38:37) Like you can say the probability that it will fund at series B with an up round. (00:38:42) So you can do whatever you want. (00:38:44) So we can think about what are those things that are really necessary for this to survive? (00:38:48) Because the thing is, I can guarantee you, won't ever have a fund returner if you invest at series A that doesn't invest at series B, likely not a down round, which is a bad sign. (00:38:58) Let's take that one data point, right? (00:39:00) But then you can actually think about what are all the data points that I now want to be making predictions about where I'm going to start to know those things more quickly. (00:39:10) Now, what's nice about that is it does double duty. (00:39:14) What is it massively increases the outcomes that you have so that you can get more into the thousand coin flip situation? (00:39:20) Because let's imagine that you have 100 companies come into partner meeting in a year (00:39:26) And you've made that, we're just talking about the one forecast right now, you've made that one forecast for all 100 companies in the space of 18 months, you're going to know for every single one of those companies, because it's public whether they do a B, how good of a predictor am I of this particular thing that really, really matters. (00:39:46) So now you start to be able to close these feedback loops faster because you're generating so much more data that we start to remove the luck and it becomes about sort of your predictive power. (00:39:56) Do you actually?