Epistemic Path Dependency, w/ Adam Harstad

We were absolutely thrilled to be able to sit down with Adam Harstad, who is basically Rob's secret weapon to dominate every year in our office Fantasy Football league! Fantasy sports is just one avenue that allows Adam to display his incredible art of storytelling through data. References in this Episode: Football Guys Forums Football Outsiders Imperial March Major Key Version The Map Is Not The Territory The Treachery of Images  Reactive Armor Zidane Headbutt Joey Harrington Article Blotto Game Ted Lasso Dart Scene(FOUL LANGUAGE) Episode Timeline: 3:30 - Adam is a HUGE fan of path dependency, SSOG and The Crying Elephant, the ups and downs of being a sports fan 17:05 - Data and Storytelling, Business data vs. Sports data analytics, 46:40 - Risk assessment, The NFL Draft, Sunk Cost examples, Adam finds his niche with Fantasy Football 1:08:10 - Single trial experiments, Calibrating predictions and models, and Super Forecasters Episode Transcript: Rob Collie (00:00:00): Hello friends. There was an old Monty Python movie titled And Now for Something Completely Different. I think that's an excellent tagline for this show, and today's guest, Adam Harstad. Nominally, Adam, is a fantasy football writer and analyst, but I think you'll see that that title or really any succinct title really struggles to describe and contain the human being that is Adam. A better term for him might be something like curious adventurer, because that really seems to be the way he approaches everything. Rob Collie (00:00:34): We went in a lot of weird and nerdy directions, as you might expect. Yes, of course, we talked about football. So if you're not into football, well, you can tune those parts out. But even there, when we're talking about football, we're really talking about the fundamentals of how to think and specifically how to think in the face of uncertainty. We also talked about the differences between sports analytics and business analytics, which of course are significant. And we revisited an old favorite topic that of Nate Silver versus elections. Rob Collie (00:01:04): I learned a lot of new vocabulary along the way, and I really must say that I felt pushed in this conversation in a good way. He was pressing us to be better, unintentionally, politely, but that's what happens when you have really intelligent, vibrant, capable people in your conversation. We all should be seeking out the Adams in our life because they pull us forward on that journey of improvement. And he's an entertaining fellow too, so let's get into it. Announcer (00:01:35): Ladies and gentlemen, may I have your attention please. Announcer (00:01:39): This is the Raw Data By P3 Adaptive podcast with your host, Rob Collie and your cohost Thomas Larock. Find out what the experts at P3 Adaptive can do for your business. Just go to p3adaptive.com, Raw Data By P3 Adaptive is data with the human element. Rob Collie (00:02:03): Welcome to the show. Adam Harstad, how are you, sir? Adam Harstad (00:02:08): I am super happy to be here today. Rob Collie (00:02:10): Well, I appreciate that. You're a really good sport. Talk about just blindly wandering into something like a volunteer. No one has had as little context for coming on this show, as we've given you, we've really given you nothing and you're still just all smiley, ready to go. Adam Harstad (00:02:27): Yeah. We got a family motto for our little kids, we wanted to create a sense of identity. This is who we are as a family and our family motto is Harstads try new things, so I'm pretty much up for whatever. Rob Collie (00:02:39): Damn, that's how this came about is I saw a tweet from you a while back that said, "Hey, if you're looking for a podcast guest..." You had this long list of things that you were down for. And I'm like, "I'm going to take him up on it. Let's see if he actually means it." And you totally did. Adam Harstad (00:02:54): Absolutely. Yeah. Rob Collie (00:02:55): I know you better than you know me because I've been following on Twitter and actually elsewhere for a very, very long time, not super closely because I'm not really that fanatical about fantasy football, at least compared to what I used to be, but you're a data guy. There's no two ways about it. And also, bring that humanity, there's so much creativity and that right brain stuff going on at the same time. And that's, to me, where those two worlds meet is just golden. I love that. Let me explain to the audience how I know of you. Rob Collie (00:03:28): And at some point it's only fair, I suppose we turn the tables and Tom and I explain to you who we are. There's random internet stalkers as far as you're concerned. Right? I first discovered you, I believe on the old Footballguys forums early 2000s and you were, I believe SSOG. Adam Harstad (00:03:36): Yeah. Rob Collie (00:03:36): So I was right. Adam Harstad (00:03:46): That was back when I was young and still somewhat of a tool, which thankfully I've outgrown. Tom Larock (00:03:51): What's SSOG? Adam Harstad (00:03:53): Yeah. Speaking of growing out of being a tool, I got on the internet when I was 16 years old and I needed an email address and Hotmail was the thing at the time. So I signed up for a Hotmail account and being 16 and extremely full of myself. My first email address was [email protected] because that was the joke. I'd say something and people would say, "What do you think you are, some sort of genius?" Then I could respond back that, "Yes, I do think I am some sort of genius." Rob Collie (00:04:20): I do. Exactly. Adam Harstad (00:04:22): That lasted for a year or two and then I realized, this is a little bit much, even for me, I'm all about owning who I am, but this is a little bit much even for me. Then I got into the University of Florida, so I changed it to some Gator, which was relatively inoffensive. And then just over time, it had gone through multiple- Rob Collie (00:04:41): Relatively. Adam Harstad (00:04:42): Relatively, absolutely. Depending on where you're from, depending on who you're asking. Over time, it went through multiple iterations where the commonality was just the initials and so, it became a running joke. People ask what SSOG stands for, and at one point it stood for stuff and then after a while it just stood for whatever you wanted it to stand for. Rob Collie (00:05:00): Now you're just SSOG. Adam Harstad (00:05:02): It's a testament to the power of path dependency. And I think, if you ask other questions about other elements online profile, it's all the same thing. It's a reminder to myself that where we are is largely a function of where we started and the path we took to get here. Tom Larock (00:05:16): That's deep. Rob Collie (00:05:17): Yeah. It's a concise summary in a way of like everyone we've had on this show, we've yet to have a guest on this show whose career was a cold shot. We're still looking for that person who set out to be something in high school that they knew what they wanted to do with their lives and they went out and they did it and they love it and they're successful at it. We're still looking for that guest. Adam Harstad (00:05:38): You should meet my wife. That's her in a nutshell, which is- Rob Collie (00:05:41): Really? Adam Harstad (00:05:41): It's always amazing to me. I can't fathom that sense of who you are at that young of an age. I still have very little sense of who I am. I'm still working on it, but she's always known. Rob Collie (00:05:52): Wow. That's got to be really interesting, as a close up example. Adam Harstad (00:05:58): Yeah. Rob Collie (00:05:59): My initial reaction is, wow, that would be really invalidating for me, but then I recovered. I'd be okay. I'd be okay. I'd be okay with my zigzag path. Adam Harstad (00:06:06): I've managed to survive for 20 years now with it. Rob Collie (00:06:10): Footballguys forums were one of the handful of places on the internet back at that time, where if you were really crazy obsessed about fantasy football, which I was at the time, you could get exposure to some really smart people in the community and their opinions. Not everything posted on those forums was intelligent or worth following, but it was amazing how, after a while, you, as a silent lurker on those boards, I almost never said anything. Rob Collie (00:06:36): There's a handful of personalities like these seven or eight avatars personas that you start to associate with subconsciously like, these are competent people. SSOG absolutely was one of those. And so now all these years later, the mystery of SSOG, the origins of the acronym unraveled. This is the first time I'd heard that, but you also had the crying elephant is that Babar? Adam Harstad (00:07:00): I have no idea what it is, to be honest. Speaking of path dependency and everything about my online personas attribute to that path dependency. I was on a forum and I needed an avatar and Google image search was the brand new thing. And I'm like, "I'm going to play around with Google image search to see what it does." I went onto Google image search, and I typed in SSOG and I looked at the page of the results. But unbeknownst to me, I had accidentally... My left hand was shifted one key to the left. Adam Harstad (00:07:29): So I actually typed in AAOF and the crying elephant was one of the first results and I'm like, "This is really cool." And I love that story. That it's serendipity, that just my hand was shifted a centimeter to the left. And now I've attached it to everything to create this consistent online brand because this is back before people were using real names and having a persistent online persona. But I really like the avatar. I think it's a little cheeky. It's a little bit fun, a little whimsical. And it, I think is mysterious. People are always wondering who is the elephant? Why is he crying? Adam Harstad (00:08:04): But I also love it because it's a complete accident, and that is a good reminder to me sometimes when I need one. Rob Collie (00:08:11): You can definitely go check this out right now on Twitter, Adam Harstad. Is that right? Adam Harstad (00:08:16): Yeah. Rob Collie (00:08:17): You're not @SSOG, but you kept the SSOG icon, the original elephant icon, the AAOF icon as the case to be. Adam Harstad (00:08:26): Again, path dependency. I tried to register SSOG on Twitter first, there is an account on Twitter registered @SSOG that has one tweet. It says, "In a jerking mood." And that's the entire SSOG account. Rob Collie (00:08:41): Are you sure that wasn't you when you were- Adam Harstad (00:08:44): I’m positive. Rob Collie (00:08:45): ...half-awake one night? Tom Larock (00:08:47): [inaudible 00:08:47]. Adam Harstad (00:08:48): That's why I write under my own name as opposed to a pseudonym. At some point I probably made a transition. Tom Larock (00:08:54): That account is suspended. Adam Harstad (00:08:56): I think so. I think so. Just because it's registered in 2009, because I was on relatively early, not super-duper early, but I've been on for over 10 years now and that one beat me on and tweeted, "In a jerking mood." And then just never logged in again. Rob Collie (00:09:10): But that describes Twitter so well, don't you think? Adam Harstad (00:09:14): Absolutely. Absolutely. And they did the alternating caps and no caps. I love it. And that's why I write under my real name. Rob Collie (00:09:20): By keeping the elephant, I was able to connect you with who you had been on the forums. Are you still on the Footballguys forums at all? Adam Harstad (00:09:28): Not so much anymore. People think there's this grand conspiracy that once you start writing for Footballguys, that the bosses don't want you on the forums as much, which couldn't be further from the truth. It's more just, people have kids, people grow older, people have less disposable time. Rob Collie (00:09:41): Let's just go back for a moment. Florida Gators, I was raised with the Gators as a religion. I betrayed everybody and went to Vanderbilt. I was the first person in my family to break ranks, but then you ended up a Denver specialist, right? Adam Harstad (00:09:54): Yeah. I grew up in Colorado. Till I was 15, I lived in Colorado and then we moved out to Florida. I wasn't a diehard Gator or anything. It's just, I lived in Florida. It was in state tuition. It was the best public school in the state, so that's where I wound up. Rob Collie (00:10:09): Also, coincidentally, because I was raised so aggressively on the Gators. At an age when you're just defenseless, you're just going to take an imprint and it's going to stick. My favorite NFL team for a very long time was the Broncos because of the same color scheme. Adam Harstad (00:10:24): Orange and blue. Rob Collie (00:10:25): That's right. I suffered through all of those terrible blowout losses. I used Apple II Print Shop to make a banner for the Broncos and hung it on the front of my house in middle school just to have them get destroyed. Adam Harstad (00:10:41): At least they were making it though. Rob Collie (00:10:42): That's right. Adam Harstad (00:10:43): It's hard to complain too much as a Bronco fan. There was a three decade span where we had more Super Bowl appearances than losing seasons. Rob Collie (00:10:50): Yeah. When you think about being a fan for a sports team, it's a losing proposition, the probability. There's only one team that's going to end on a high note, that's it. There's one team. Take the number of teams in the league, divide one by that that's your percentage chance of ending the season happy. I just think the expected value of being a fan of a particular team is actually pretty low. Even the good years where you win the championship, I just don't know that that really makes up for the down. Rob Collie (00:11:19): I've become more of a sports fan for that reason. A little bit more mercenary. Here we go. Tom Larock (00:11:25): This isn't true. That's really extreme. First of all, you could say that about any sport, but what you really mean to say is only one team and so seasoned on the wind, but there's no way you're going to tell me that a Cleveland Browns fan didn't end last season happy to beat Pittsburgh in the playoff game after how many shitty seasons. They ended happy. They didn't win it all, but they're certainly happy. Rob Collie (00:11:50): Think about though, how the ground needed to be prepped for them to reach that level of happiness about mediocrity. Tom Larock (00:11:58): That's fair. Rob Collie (00:11:58): They had to have decades of misery. I just- Tom Larock (00:12:01): You’re talking to a Patriots fan; I know decades of misery. Rob Collie (00:12:05): I know. I just don't think that being a sports fan for a particular team was a buy and hold strategy. You inherently need to time that market if you want to win. Tom Larock (00:12:13): Buy and hold is interesting because as a child, my friends and I, we would pick one of the worst teams to start rooting for because we knew in 20 years, they'd be good and then we could say, we were a fan way back. I'll go get my Vikings jersey right now just to show you. Rob Collie (00:12:32): Any year now. There's this, and it's a well-known human tendency that we judge experiences predominantly by the very, very end. They've done studies where they'll play people, this beautiful symphony and then right at the very end, they end on a sour note and people will say it ruined the entire symphony. There's 30 minutes of music and it ends on one note. And I just think that's the wrong way of looking at it. Rob Collie (00:13:00): You had 29 minutes and 50 seconds of just the sublime transcended experience and 10 seconds of something bad and you end your season on a loss, but the season, isn't just the last game of the season. It's all the anticipation leading up. It's everything you do during the course of the year. A lot of troubles in life are just a matter of perspective and a matter of attitude. Tom Larock (00:13:20): I want to hear one of these now. Is it just a sad trombone at the end? Rob Collie (00:13:26): [inaudible 00:13:26]. Tom Larock (00:13:36): Because I'd just be sitting there laughing. This is the greatest thing ever. Rob Collie (00:13:40): The meta of it would immediately just like crash over me. Like, are you kidding, you designed an experiment like this? Adam Harstad (00:13:48): I think it doesn't resolve, you have the progression and it's supposed to resolve on a certain note and they're probably a half step flat. And I don't know if you know that much about music, but if something doesn't resolve, it creates this very real physical tension because you're anticipating it and you're anticipating and you're waiting for it. And it's really a cruel thing to do to someone, but at the end of the day, it doesn't change the first 29 minutes of the symphony. Tom Larock (00:14:12): Tomorrow night I'm going to Tanglewood to see John Williams in the movie night. Now he's 89. I'm pretty sure he'll still be alive for tomorrow, so that's good. But now I'm just thinking he's going to play the Star Wars theme and at the end of it, it's just a sad trombone. I'm just going to be sitting there laughing at the end of all these songs. People are going to be like, "What is wrong with that person?" Rob Collie (00:14:36): All right. I don't know anything about music theory. I freaking love music. I'm always, always, always deep, deep into music and stuff that I appreciate is usually like on the more complex end and I don't know anything. Let's talk about this for a moment. I heard the Imperial March one time in the major key instead of the minor key. The minor key Imperial March is very foreboding. The major key version sounds like this upbeat, let's go have fun. It's like a clown parade. Explain it to me like I'm five minor key versus major key. Rob Collie (00:15:13): Have we come to the point where I'm going to understand this in my life? Adam Harstad (00:15:16): The simple version is minor key sounds sad, major key sounds happy, but that's too simplistic. There are happy songs written in minor keys. There are sad songs written in major keys. It's been a lot of years since I have last taken any music theory courses and that's unfortunately some of the knowledge I've lost during my life. Rob Collie (00:15:32): Well, that's okay. Path dependency again. Right? Adam Harstad (00:15:35): Right. Rob Collie (00:15:35): I've also heard though that it's cultural, certain cultures will actually find the major key to be the sad one and vice versa. Adam Harstad (00:15:43): Yeah. Most of our experience of music is just based on historical context and our exposure to music. What sounds normal. A lot of our enjoyment of music is actually anticipation. There's pattern recognition and there are certain patterns that are common through music and the brain really loves anticipating those patterns and then when the patterns resolve in the way that we expect them to, the brain rewards us with a jolt of pleasure like, "I knew that was going to happen. That was very enjoyable and pleasant." Adam Harstad (00:16:10): Sometimes if the patterns resolve in ways, we were not expecting brain results us in a jolt of pleasure because this was fun and new and exciting. I'm eager to see where it goes next. And so, our experience in music is heavily shaped by the patterns and our experience to music prior to that. We can listen to a Western pop song and we're very familiar with the genre, we're familiar with the tropes. We're familiar with, we're going to go first course first. Maybe sometime we'll have a coda. Adam Harstad (00:16:40): There's the common building blocks of music and artists will change the order of them, they'll change the way they're put together. They'll change the makeup of those blocks, but they're building with the same blocks, whereas we can listen to Eastern music and it just sounds very foreign to us because we don't have that language. We don't have that fluency in it yet. Tom Larock (00:16:58): It's such an interesting topic, almost like this anticipation. I totally see that. Rob Collie (00:17:02): And there are some quirky misdirections in music that you wouldn't necessarily expect, but that are incredibly fun when they spring them on you. It's not just personality that I've been drawn to that it kept me reading your stuff over the years. And in fact that wasn't at the beginning at all right. It was the quality of the insight and most of it is data-driven. Most of it is analytical. It's not just about the evidence that you collect. It's also about the way that you process it that I have found valuable over the years in trying to defeat my friends and colleagues in a silly game. Rob Collie (00:17:35): How did you discover your passion for data? Adam Harstad (00:17:38): It's funny. I came at it obliquely. I've got a very strong left brain, classically left brain skills, math and seeing patterns and interpolation, things like that. But I think my love of football and my love of fantasy football has actually always been more narratively driven. And I like to say first and foremost, I'm a storyteller. And it's funny you say I'm a data guy. I know data guys who think I'm one of those narrative guys, narrative guys think I'm a data guy, I'm in that liminal space between. Adam Harstad (00:18:08): For me, it's all about, I have these pressing questions and I think the questions are the interesting part. And I'm looking for any insight, knowledge, any edge I can get to answer those questions. And getting back to path dependency, when I was getting into fantasy football in early, early 2000s, that was really the unexplored space. There's a website called Football Outsiders that was just launching and they were actually looking at data and they're saying, let's compare every play in this situation to every other play in the situation. Adam Harstad (00:18:40): If you ran for four yards in first and 10, was that a good play? Was that a bad play? Nobody had ever really dug into it. And so, that was the edge. That was where the edge is. And over time as analytics has gotten to be a bigger and bigger part of the sport of football and the hobby of fantasy football, I find often the edge is in this, not really contrarian to analytics, because I think analytics is good and useful and right. Analytics is a way of approaching problems by looking at the data and that's a very, very useful tool. Adam Harstad (00:19:13): But I think that there's some conventional wisdom that should apply to analytics that has gotten lost along the way in the rush to mine more data. So for me, it's never been specifically about the data. It's been about the questions and the answers. And for most of my career, the data has been the best way to reach those answers, but it's not the only way. It's more about finding the appropriate tool for the appropriate situation. Rob Collie (00:19:37): I think I'm starting to understand why I felt this remote kinship with you asymmetric, because that's how the internet works with lurkers and publishers. Our team is growing really rapidly at our company and we're a remote team. We're all over the country, even though we're full-time employees, it's a good problem to have, but it's become difficult to keep track of who everybody is. Name, face, okay. But what about personality? It's so hard. Yesterday we started collecting information to make flashcards for everybody. Rob Collie (00:20:11): Everyone had just to pick a "superpower" or something like that, but not something super professional, but like storyteller. I picked storyteller for me, folksy storytelling would be one of my, "Look out, he's going to do it." And then the football, I'm a data professional. There's no two ways about it. I worked on data tools at Microsoft, I worked on Excel, I worked on power BI, I'm CEO of a data analytics, business intelligence consulting firm. Yes, data professional. But that all started for me in the late '90s with the same sorts of things as what you're talking about. Rob Collie (00:20:45): Like how do I actually approach this game, fantasy football? How do I approach it more effectively? I stumbled upon the original value-based drafting article years ago, which put the different positions into perspective in terms of relative value. And I was just like, "Ugh" It's like the holy grail. It was this magical unlocking moment and what followed was just reams of spreadsheets and coding and all kinds of things. It was the only times, even though I was a software professional already, I was already working at Microsoft. Rob Collie (00:21:15): I was never really that into it. It was one of the few times that I was actually into it was when I started doing these sorts of things. Another thing that you wouldn't have the context for is that on this podcast over and over and over again, we talk about how we observe even, the value is in the hybrids of different centers of mass. In the business world of data, it's long been like these completely separate universes where IT was doing some data stuff and business was doing some data stuff and they almost never collaborated. Rob Collie (00:21:51): And really the software was built to create this divide in the old days, but now there's this rise of these IT/business hybrids. And again, it comes back to all these sorts of things that you were saying conventional wisdom is a shortcut for saying all kinds of things, all kinds of developed expertise and wisdom that you can't just set aside when you're using data. You need to integrate the two. IT can come into a business situation and bring all kinds of technical skill, but they've got a complete blank slate when it comes to the business knowledge of what's actually going on. Right? Adam Harstad (00:22:28): Right. Rob Collie (00:22:28): It turns out that you can't have one or the other, you have to have both. And that just keeps coming up on this show for good reason so I think there's a fundamental truth that we're experiencing there. Adam Harstad (00:22:39): Yeah, for sure. I think where fantasy football at is gotten a little bit too disdainful of other forms of received wisdom. There's this idea that what the data says is true, which is usually the case, unless you're making mistakes, analyzing the data, which is very, very easy to do, but there's also the sense that what isn't coming from the data is not necessarily false, but suspect. And as an example, there was a recent debate about whether throwing two running backs is a positive or negative for NFL offenses. Adam Harstad (00:23:11): And the data guys looked at it and they said, if you look at the expected value of every play, passes to running backs are less valuable than passes to wide receivers, which is true, I'm sure that's exactly what the data says. They're smart guys. They know how to analyze it. I'm sure that they're controlled for the appropriate confounders, various other pitfalls, but at the same time, I'm looking at it like, that may be so, but if you look at the best offenses of the last 15 years, disproportionately, they've thrown to their running backs a lot. Adam Harstad (00:23:40): You look at Sean Payton's New Orleans Saints throw to their running backs more than any team in the of football, and it's hard to find any offense in history that has had as much sustained success as the 2006 to 2020 New Orleans Saints. Bill Belichick, New England Patriots, you have James White, Shane Vereen. They historically have had a back whose only job is catching the football. Danny Woodhead was there, and it's hard to look at these and say, throwing to running backs is a bad play and yet a lot of the best offenses in history do it a lot. Adam Harstad (00:24:13): A lot of the worst offenses in history do it a lot too. I don't think it's a silver bullet, but it's hard for me to reconcile the idea that this is just outright bad with this observation that it's so often successful. And so, to me, the received wisdom is maybe we have reasons to be suspect of passes to the running back, but I can't just outright declare it bad, but that's not coming from data. That's coming from anecdote and observation, which is a powerful tool for receiving insight that I don't think is necessarily reckoned with enough. Tom Larock (00:24:45): On that, what I would say, and I got a whole lot I want to talk about when we talk about bias and things like that, and I just followed you on Twitter so I could retweet your thing about selection bias. I started fantasy football in the mid '90s, roughly. Well, I think we all did when it was really up and coming and no offense, but I'd never heard the Footballguys until today. I knew the Football Outsiders and I knew a handful of websites and this was before even ESPN or CBS Sports was offering this data, so I've been there. Tom Larock (00:25:16): What you're talking about just now with the throwing to the running backs, what hit me was all the data that you're examining is historical and you have no idea of what's going to happen next. You can say, everything we do in fantasy football, all we're doing is managing risk. I have a risk of scoring zero points, how do I make sure that I'm at least scoring something and that's all we're doing. We're just looking for a way to make sure we're producing some value on the field, so to speak. Tom Larock (00:25:48): Anyway, let's look at, if you could, as an example, before the 2007 season, when the Patriots had Moss and Wes Welker, nobody had any idea of just how many points that team was going to put on the board that year, because historically it just didn't happen. Maybe there was an offense from the 40s that ran something similar, but probably didn't have the same numbers generated. But when you talk about Belichick and a lot of coaches in the league, everything old is new again. Tom Larock (00:26:20): He just says, "All right, what talent do I have? Where would they excel?" This type of a formation. It's not really hard. They just look and say, "It's fairly basic. You need to do these things to be successful. Wes, go into the slot, run that way. Brady might hit you because Moss is just going to go de..." My point is simply, you can do all this analysis, at the end of the day you still don't know what's going to happen if somebody gets hurt. And to me, like I said, it's just managing the risk of scoring zero points. That's how I look at it each week. Tom Larock (00:26:52): I have a risk of scoring nothing, I can actually in this league be negative. How do I avoid that? Adam Harstad (00:26:57): One of my favorite articles I wrote was actually, you guys are probably familiar with the phrase, "The map is not the territory."? Rob Collie (00:27:02): No, I've never heard it. Adam Harstad (00:27:03): It's this idea that a map obviously is not the territory that it represents. It's a map. If you draw a mountain on the map, it's not like a mountain is going to magically spring from the ground and it's a representation. And I do a lot of historical modeling and that's a lot of my process is I will look at historically comparable players. Ja'Marr Chase comes in, what's the history of highly drafted rookie wide receivers? And from that, I'll create my baseline expectations. You need your prior and you can adjust your prior from there, but it's good to have that strong prior to start. Adam Harstad (00:27:38): But the problem is, let's say that one of the historically comparable players for Ja'Marr Chase is Larry Fitzgerald, that's going into the mix. If Larry Fitzgerald had torn his Achilles tendon as a rookie and never played another snap, the comparable players for Ja'Marr Chase would be worse, but his prospects wouldn't be any different. Realistically, it's not like Larry Fitzgerald tearing his Achilles would make Ja'Marr Chase more likely to tear his Achilles. Adam Harstad (00:28:05): Similarly, if someone like Charles Rogers who was a colossal bust had gone on to become a Hall of Famer, the comparable players to Ja'Marr Chase would look better, but Ja'Marr Chase's prospects wouldn't change at all. And it's something that I think you really need to reckon with when you're doing this kind of historical modeling. You need to remember that the map is not the territory. Every player in this situation has done poorly before, it doesn't mean that this player is going to do poorly now. Adam Harstad (00:28:28): But the full quote is, “The map is not the territory, but it has similar features accounting for its usefulness." If it's drawn correctly, it will resemble the territory enough that you can use it to navigate and it'll get you where you need to go if it's a good map. But the famous René Magritte has the treachery of images. It's the picture of the pipe and underneath he writes, "This is not a pipe." And everybody gets really mad because they're like, "Of course it's a pipe. Of course it's a pipe." Adam Harstad (00:28:54): And he said, "Okay, well, can you stuff it? Can you light it? Can you smoke it? No, it's not a pipe. It's a picture. It's not a pipe." The map is not the territory, but it shares similar features and that accounts for its usefulness. Rob Collie (00:29:06): We still need maps. Just be careful how you use them. Adam Harstad (00:29:09): Lewis Carroll wrote once about a country that created a map with a scale of one mile equals one mile, but then they didn't find it that useful, so now they use the territory as its own map and it serves almost as well. Rob Collie (00:29:20): Yeah, it's perfect actually. Tom Larock (00:29:24): That sounds more like a Steven Wright joke. Rob Collie (00:29:26): Yeah, it does. Sports, data, sports analytics. I think it draws a lot of the same personalities to it that business data draws, but the similarities between the two as domains, as professional activities, or even as hobbies, they don't actually share that much. There's some really, really, really important differences between the two. One of them is that sports is adversarial. It's a hundred percent adversarial. It's one team versus one team or one player versus one player. And the business world isn't like that. The business world inherently, it's competitive, but it's much more collaborative. Rob Collie (00:30:09): People you're interacting with are your customers, not your direct competitors. And so, that changes everything. There's just so many things that happen. Like when we were talking earlier about throwing to the running back, for instance, one hypothesis we can form is that if you're effective at throwing to the running back, you maintain an information advantage over your opponent. You keep them guessing, you're unpredictable in a way that works to your advantage. Being unpredictable doesn't work to your advantage in business. Rob Collie (00:30:40): There's nothing to be gained by, "You didn't expect that." It's just not like that. Tom Larock (00:30:46): You mean every time Google kills a product, that's just not helping them? Rob Collie (00:30:50): No one expected Microsoft to buy ProClarity and basically put an end to it back in the day and that was just good fun. Kept things interesting. Adam Harstad (00:30:58): Kept everybody on their toes. Rob Collie (00:30:59): Yeah, that's right. You never know. We might buy the number one front end for our server and basically retire it. Anyway. They don't do that anymore. That's the old Microsoft that did that kind of thing. Adam Harstad (00:31:13): Monopoly rules. Rob Collie (00:31:14): Yeah, that's right. And also, software engineers running the show. Anyway, we're not going to talk about Microsoft too much today. The other thing about sports is that almost everything that you would do analytically with it is predictive in nature. Whereas in business, not everything, there's still absolutely predictive stuff that needs to be done in business and we do some of that for our clients. But like, let's say if I was speaking to the other audience, the sports analytics audience, who's never worked a day of business analytics. Rob Collie (00:31:45): The analogy I would give you is, what if you only found out the score of the game, that's all you learned was the output like, "Shit, we lost again. Bad this time. 31 to 10." But you didn't get to watch the game. You don't get any box score and we're the ones that use all of this incredibly fine grain data that's happening in the business to eventually explain why things are happening the way that they are. Even just knowing where you are, knowing what's actually happening. Rob Collie (00:32:12): There's a joke in our industry, business intelligence by bank account, which is you get to the end of the year and you're like, "Hey look, how much money is in our account. Way to go team." Or, “Damn. Terrible year. Everybody just grit your teeth and try harder next year." That's reality. The lights are out by default and it's a tremendous oversimplification, but at the same time, there's a lot of truth to it, which is, we go around turning the lights on, even just explaining what has happened. Rob Collie (00:32:41): Typically, we don't want to be like on a one-year delay to understand what's happening, we want to know early in the month, what you're trending towards and why, so that you have an opportunity to make changes. It's all about forward changes. So many differences about sports analytics versus business analytics. And yet we're all moths drawn to similar flames. I watched the skills that you've developed over the years as I've been watching you, Adam, are very different than the skills that I require in the business that I'm in on a daily basis. Rob Collie (00:33:13): There are outlier applications for a lot of the things that you do in business. Absolutely, but it's like the average task, the average analytical question in sports versus business is really completely different. Adam Harstad (00:33:27): Well, and I think a lot of it too, is that in sports, the whole possibility space is so constrained. If you have an offense at the one-yard line, there's basically a hundred possibilities for that play. They can gain zero yards, one yard, two yards, three yards, so on and so forth. Whereas, if you have a startup at the business equivalent of the one-yard line, the possibility space is just so much larger. They can succeed, they can fail, they can succeed and fail in old ways. They can succeed and fail in new ways. Adam Harstad (00:33:56): Really, everything's on the table, but sports because it happens in a clearly defined arena of competition, under rules that are laid out to all the participants in advance and everybody has an opportunity to study, maybe some people are more familiar with the rules than others, but they're all constrained by the same rules. The world works in a very predictable and ordered way in a way that the real world outside of sports just doesn't operate. Adam Harstad (00:34:21): And so, yeah. It doesn't surprise me that there's not that much carry over because it's such an arbitrary space to analyze. People talk about sports as a metaphor for life, and maybe it works as a metaphor, but it's not really a great analog Rob Collie (00:34:34): At the heart of it, let's get metaphysical nerdy. Adam Harstad (00:34:36): Totally. Rob Collie (00:34:38): In the history of warfare, which has actually a lot in common with sports, whether we like it or not, that adversarial nature, the only constraints are the physical world. At any point in time, there's never been impervious armor. The armor of the day has always been vulnerable to the weapons of the day. There's never something that, you could just build, it was invincible. Of course, armor of today probably could stop everything from 2000 years ago, but at that particular point in time, and why is that? The same materials, right? Rob Collie (00:35:09): Same materials, same manufacturing, same technology is available to both weapon makers and armor makers. It's just that the weapon, only the weapon "knows" where it's going to hit. There's an information advantage that the weapon has. The armor has to be prepared for everything, but the weapon gets to pick whatever spot it happens to hit. There's an information advantage for the weapon and like in football or really in sports, going back to the thing we were talking about, the being unpredictable, but still effective seems to be like a crucial element of a well-designed offense. Rob Collie (00:35:42): And guess what? There've been very, very, very few defenses in the history of the NFL that are capable of shutting out their opponents, very similar to the armor and weapon thing. Information conveys a very, very specific advantage in adversarial situations. Of course, information also conveys an advantage in business. I can't put this topic down. You seem smarter than I am, so let's see what you think of this. Adam Harstad (00:36:05): I seem that way, for sure. SSOG, man. Projection. It's a careful projection of certain appearances. Yeah, I'm thinking about it. I think a lot of it's probably an incentives issue too. I don't know that there's the incentive to build impenetrable armor, wholly impenetrable armor like there is to build unstoppable weapons. If you look at history and you're mentioning that war is very zero sum, the societies that survived and the cultures that survived are the ones that aggressively expand it. Adam Harstad (00:36:38): If you had a hypothetical city state that just wanted to just sit in their city and never expand territory, eventually they're going to be overrun. And so it's almost like the rules of the conquest favor the weapon making over the armor making. You need armor that's just good enough to allow you to use the weapons that you've created. I completely agree that information asymmetry, especially in zero sum competitions is very important. One game theory concept I go to a lot is called a blotto game. Adam Harstad (00:37:11): And a blotto game is a game between two people where you each have a certain number of resources. Say you're a general and I'm a general and we're contesting three battlefields. And we each have five units that we can send to those battlefields. And whoever sends more units to a specific battlefield is going to win that battlefield. And whoever wins more battlefields is going to win the war. You can allocate your units however you want. Adam Harstad (00:37:33): You could send all five units to one battlefield, which will guarantee you that you conquer or at least hire that battlefield, but then you're leaving the other two battlefields undefended and the other general will almost certainly win the war. That's definitely a losing strategy you could send and there's also, you could do three, one and one, you could do two, two and one and you can evaluate all the possibilities and some of them will be Pareto optimal, some of them will be completely dominated by other strategies. Adam Harstad (00:37:58): There's a strategy that will always lose to another strategy or at worst tie it. And you can study these blotto games and there's variants on them. Say one side has more resources than the other. Say that some battlefields are worth double points. It's different if you're just going to play at once or if you're going to iterate it over and over again. But the best advantage you can have in a blotto game is having more resources than the other guy. If you have 10 units and I have two units, it's going to be trivially easy for you to win. Adam Harstad (00:38:26): And then the second best advantage you can have in a blotto game is knowledge of what the other guy is doing. Because if I know your strategy, it becomes very easy for me to just pick the strategy that defeats it. And in zero sum endeavors, fantasy football, among other things. I find a lot of instances that if you look at it, it really is just a more expensive version of the blotto game. Even football itself, I would call something of a blotto game. An offense has five receivers. These are its units. And it's trying to capture territory. Adam Harstad (00:38:58): It's trying to get a receiver into an unoccupied space by the defense so that it can complete a pass and progress the football, and then the defense has its seven defenders and it's trying to defend this territory. And really there are usually on most plays seven guys in coverage versus five guys in pass patterns and yet the offense wins more than the defense. And the big reason why is because the offense knows what it's going to do, it knows where it's going. Adam Harstad (00:39:23): It has that informational advantage and especially quarterbacks, Peyton Manning famously could read the defense before the snap and know what it was going to do. And so, it was much easier for Peyton Manning to win that particular blotto game than an average quarterback, because he had this fore knowledge of how the defense was going to be deploying its troops, what battlefields it would be contesting and which battlefields it would be leaving relatively open. Rob Collie (00:39:43): Have you been exposed to the concept of reactive armor? Adam Harstad (00:39:47): No, I haven't. Rob Collie (00:39:48): To me it's genius and it's only something that the Russians would have come up with and they had it back in the Cold War. If we look at the information advantage that the giant metal dart fired from NATO tanks has, it only has to worry about the place where it hits. All of its work is concentrated on this little square centimeter or whatever of armor that it impacts. And all that other armor is useless. All that weight that the tank's been carrying around, except for this one little column of metal that's in front of the dart, all the other stuff doesn't participate in his interaction at all. Rob Collie (00:40:27): And the Russians said, that's the problem, so when we get hit with a dart, we're going to have an explosive go off within our armor and slide the armor sideways in that section so that it's constantly feeding fresh metal into the interaction. It's crazy. And when the iron curtain came down, when the Berlin wall fell and NATO took some modern Soviet tanks out and tested them to know if the darts would go through, this armor stopped everything. And what was the predictable NATO response given the way that we think? Rob Collie (00:41:04): Just make the darts longer. But we see this, the defenses that are able to react like in football, more nimbly and quickly to what they see in front of them are the ones that are... There's no overpowering the offense when the offense has the information advantage, even though you have more players. Adam Harstad (00:41:25): There's one aspect of the defense, the pass rush actually is the whole situations are reversed when it comes to defending the quarterback. It's the offense that is on defense so to speak, that's reacting. And it's the defense that gets to choose, we're going to pass, rush these players. They don't know who's going to be rushing. We're going to rush from these angles, we'll be doing stunts to try and create confusion. So, for the most part, on a grand level, of course, it's the offense that's reacting. Adam Harstad (00:41:48): But it's interesting that there's also that sub competition at the same time where the defense has the first mover advantage. And if you look at player salaries, they largely reflect that. The players who get paid the highest are the wide receivers who have the informational advantage against defenses and then they're the defensive ends who have the informational advantage against offensive lines. Rob Collie (00:42:11): Interesting. Tom Larock (00:42:14): Every sport is action reaction. Adam Harstad (00:42:17): Yeah. Tom Larock (00:42:18): All right. Every sport is action reaction. What I find interesting about football is, and it's probably because it's easier to see a mistake in football, I think than in other sports. And to me, football is always, maybe it's because I hang around Patriots more often, and this is why I watch it's about just limiting mistakes. You already know, Peyton Manning, he's already figured it out and he's already got a play. He already knows what to do to get a few yards. Your job is to make sure he only gets those few yards. Tom Larock (00:42:51): So you limit the mistakes that you're about to make, and you keep the play in front of you. And I think when it comes to other sports, I think baseball, the obvious mistakes, somebody falls down, but there's a lot of base running mistakes. Like I didn't take the extra base like I could have. Things like that, that you don't really see unless you're really in tune with the game. Basketball is the same where mistakes are made. The obvious one, I dribbled off my leg, but the not so obvious one is you didn't set a good enough screen. Rob Collie (00:43:18): Well, the consequences. The consequences are so high in football. A mistake in football can have ungodly implications. Whereas the worst thing that can happen in a basketball play is the other team scores, three points, four points. They score something like 4% of their total for the game and then the slate is clean again. Adam Harstad (00:43:34): Well, by that standard it would be soccer. That's the worst because the consequences of mistakes in soccer are catastrophic. Rob Collie (00:43:39): Yes. I concur. For example, headbutting the opposing player in the final minutes of the World Cup final, because he insulted your sister or something. To me that is the most leap off the couch moment in all my sports watching history- Tom Larock (00:43:57): Really? Rob Collie (00:43:57): ...was when Zidane headbutted that guy and got ejected. Tom Larock (00:44:00): Not when Tyson chewed Evander's ear off? Rob Collie (00:44:05): No, because you know what? Two whole nations hung in the balance at that moment. Who cares about two guys punching each other? This is national pride. Tom Larock (00:44:15): He wasn’t punching him, Rob, he chewed his ear off in the rain. Rob Collie (00:44:19): That's fine. Zidane's headbutt still takes it for me. Tom Larock (00:44:22): No, that's fair. Yeah, that was a remarkable moment. Adam Harstad (00:44:26): I Agree. The consequences of a mistake. What's the max mistake? Tom Larock (00:44:30): In business it's the same thing because you want to mitigate risk. There's a lot of parallels where you're going to do these things and you know that say, Excel, isn't going to ship in time. What's the risk that we're after falling behind other competing products? We can do three of the four things in order to ship on time. The one thing that we're not going to do is going to be the least costly for whatever, but that's how the world works. It's not always about being perfect, it's about making sure that your shortcomings, your mistakes are limited. Rob Collie (00:45:04): Adam, you said that at the beginning that you have a saying at your house, Harstads try new things. I don't think we've ever really codified it, but we like to say Collies play offense. In business, it makes sense to play some defense for sure. Defense is also like the luxury of businesses that are already successful. On the way up, you can't defense your way to the top. You've got to offense your way to position of value with your customers. Rob Collie (00:45:34): We don't have to talk about it as an adversarial conquest thing. You have to create something in order to stand out, you have to create something in order to change the world. I think business is inherently again, until you reach the top and become the monopoly and then suddenly you're like, 'Yeah, build walls, build walls." Except for those rare cases, it's always about create. It's always about leaning forward and being on the creation side, the offensive side of the game. And even as a life philosophy, members of my family, my extended family, they play the game of life. Rob Collie (00:46:06): You could capture their strategy as, "Avoid disappointment. Don't do that. Don't live like that." In football if you wanted a quarterback that never throws any interceptions, I'm your guy. You can hire me. I'll go out there and spike it into the ground every single play and collect my money. You want a zero interception quarterback. Tom Larock (00:46:27): I'm the guy. Rob Collie (00:46:27): I am the optimal quarterback for you. I'm pretty sure that even at 47 years old, I can get the ball into the ground before the defense gets to me. Tom Larock (00:46:35): Just three QB sneaks. Rob Collie (00:46:37): No, no, no sneaks. That's dangerous. No, no. I'm not going to be... Mm-mm (Negative). I'm going to be running away from everybody and throwing it into the ground. Adam Harstad (00:46:44): It's really interesting you bring that up because there's a lot of research now that everybody used to think of sacks as a defensive line offensive line thing, but there's a huge body of research at this point that sacks really are a risk tolerance thing for a quarterback. Some quarterbacks will accept more pressure trying to get a playoff and others are more risk adverse and they'll get the ball away quicker. And so, the famous example, I use at this is, there are two quarterbacks in history who have had two seasons in the top 10 all time in soccer percentage. Adam Harstad (00:47:13): One of them is Dan Marino, very famously able to get the ball out quickly and avoid sacks. And the other one is Joey Harrington, who is very famously one of the biggest quarterback busts of all time. Basically nobody has avoided sacks as well as Joey Harrington avoided sacks. And that wasn't a good play. Tom Larock (00:47:30): Because he's not playing? Adam Harstad (00:47:31): Because he's not playing. He's unwilling to accept any risk at all. As soon as the pressure got anywhere near him, the ball is in the stance and his offense was horrible because they couldn't gain any positive gains because he was unwilling to accept that risk. And he wrote a really interesting piece after he retired about how, "My career was not a failure." Adam Harstad (00:47:50): And he talked about, "When I was in Detroit, I was asking my head coach for permission to take risks. And I just never felt like he was behind me and he was willing to let me fail. And so I was always too scared of something bad happening to really try to make something good happen." But yeah, if you look at the quarterback performances that correlate to quarterback quality, things like interception percentage are basically meaningless. Adam Harstad (00:48:11): At the very extremes, good quarterbacks tend to throw slightly fewer interceptions than bad quarterbacks, but it's not a very pronounced difference. The big difference is that good quarterbacks make a lot more positive plays. They get a lot more yards per attempt, they throw a lot more touchdowns and sometimes they'll have some negative plays mixed in there as the risk that they're taking. It's that level of risk that they're willing to tolerate to get that offense. Rob Collie (00:48:35): Let's go back to Joey Harrington and Detroit for a moment because when we were earlier talking about let's model the history of highly drafted rookie wide receivers, and I was going to make the joke, but it's not just a joke, there was some truth to it. Well, part of the decision tree needs to be well, are they being drafted in Detroit? And they did. They burned a lot of draft capital on highly touted players. And only one of them, as far as I know, really even came close. Megatron. He was the real deal. Rob Collie (00:49:05): But all those other people, how many people did they draft in the first? It was at least two or three, right? And none of them went anywhere. Adam Harstad (00:49:11): I think they took four receivers in the top 10 in five years. I think that Charles Rogers, who was one of the biggest busts of all time. Roy Williams, the wide receiver, who was okay and they traded him to Dallas and they recouped a lot of his costs. Mike Williams, who was another big bust and Calvin Johnson. Four in five years, and it became a running joke. But the thing is the fourth one was Calvin Johnson and he was probably the best pick of that era. Adam Harstad (00:49:38): There's a lot of sunk cost fallacy that we've tried this three times, we've tried drafting wide receivers and it hasn't worked out. I think a lot of people would have cut their losses and said, "I won't deal with that ridicule of picking another one." But to their credit, they stuck to their board and they said, "This guy is the best prospect on the board." And a lot of drafts success and failure, I think far more than people would believe is just randomness. Adam Harstad (00:50:02): Any team that had been on the clock at the number two pick when Charles Rogers was there would have taken Charles Rogers because he was the best wide receiver prospect, maybe one of the best ones of all time. Possibly the best- Rob Collie (00:50:13): Maybe. Adam Harstad (00:50:13): ...wide receiver prospect of all time. And everybody thought that. It wasn't a contrarian pick. That was the chalk pick and it's just Detroit happened to be the team that had that number two pick. Rob Collie (00:50:23): Yeah, but I wanted to explore the possibility that he might've succeeded elsewhere. We were just talking about Joey Harrington not having the support of his coaches. There's clearly been something culturally rotten with the Detroit franchise for a while, or at least that's an easy narrative to paint and we don't actually know, we don't get to run the experiment. The Charles Rogers experiment. We only got to run it once with one set of variables and that's in Detroit in that era. I agree with you though, on the sunk cost thing. Rob Collie (00:50:53): Being able to distinguish the difference between a sunk cost situation and a pot committed situation is one of the hardest and most valuable things to do in life. Sometimes you can cut your losses and that is the right move. It's silly to think of it that way when you're making picks, it would be really silly to think that wide receivers are terrible, we should never do that again. That would be a bad strategy. Let's drill in for a moment. Rob Collie (00:51:18): In poker, there are decision points in a game where you only have a 20% chance of winning the pot, but it still makes sense for you to continue and invest more money because the value of what's in the pot is so great, the expected value is positive. The one time you win out of five or whatever, will pay off so heavily that it doesn't matter. It pays for all the losses. Sometimes you have sunk costs. Rob Collie (00:51:42): Sometimes it's like, no, no, just because I threw $80 in there doesn't mean I should throw another 20 and distinguishing between those two circumstances, I think there's a lot of that in business, a lot of that in life is like, which situation am I in? Am I on the road to potential success and I need to put that next piece of incremental investment or effort in? Versus, no, let's not keep chasing a bad bet. There's no formula for this. There's no formula in life for being able to distinguish between the two, but the better you get at it, the more successful you're going to be. Adam Harstad (00:52:15): And that's another situation where poker, it's much easier because the possibility space is so constrained. In poker, you can just do the math and you can say, I'm pot committed or that's all sunk costs. There's a right answer in poker. And that right is knowable. The range of possible outcomes in investing a stock is it's worth nothing or it's the next Facebook. The range is so huge. There's no way to know in advance. That's another one where I like these contrived arbitrary games with their clearly defined rules, because I think they set themselves up to analysis so much more nicely. Rob Collie (00:52:49): Well, and they also set themselves up to be great metaphors for the more complicated playground of real life. I think they very purely, sometimes anyway, very purely participate in the forms that we see with infinite noise in the real world. It helps me as a learning tool, but yeah, in order for it to even exist, you have to constrain it like you do in sport. We can't let this go without mentioning the haiku. My personal favorite interaction with Adam on Twitter was when we were, I think you started it, Adam. It wasn't my idea. I participated. Rob Collie (00:53:21): I made my one contribution, but there was a whole thread years ago of football haiku, which is exactly what you expect. An analytically minded football Twitter account, we're going to start writing haikus. That's exactly what everybody would expect. Adam Harstad (00:53:38): That's why I say, the narrative guys think I'm a data guy and the data guys think I'm a narrative guy. And I always say, "I'm just a storyteller." Rob Collie (00:53:45): You're doing it right. If either extreme is painting you into the other corner, I think that's a sign that you're over the target. You're taking flack from both sides. Well, that's where you need to be. Adam Harstad (00:53:56): In general, I tend to think the things I believe are correct, because if I didn't think that I wouldn't believe them. Rob Collie (00:53:59): That's really funny. Isn't it? And that's strange. Adam Harstad (00:54:03): Yeah. It's weird how that works out. Rob Collie (00:54:05): You could almost extrapolate that to everyone. Everyone works that way- Adam Harstad (00:54:09): It's true. Rob Collie (00:54:10): ...which is really frightening, and validating at the same time, I suppose. It's all about uncertainty, navigating uncertainty. I think that's the thing that we were talking about with the NFL draft and things like that. No one really knows anything. No one knows what's about to happen. I love that at P3, now we're up to a 14-person fantasy football league and advertised it to the team as Basket of Virtual Volatile Assets, BVBA, that's what we're really calling it. That's the game we're playing. it happens to use football, but it's not really fantasy football. It's BVBA. Rob Collie (00:54:47): And most people who are playing with us have played before, but there's a few people who haven't and so, I'm putting together a primer for them. My PowerPoint deck is in progress with the animations and everything. And the very first slide in this is going to be a collection of predictions for the year, rankings, projected rankings of players and which teams are going to be good and all that. And then there's the giant word, ha-ha, over the top. It's all wrong. It's not even going to survive a week. And every year crazy things happen. Rob Collie (00:55:20): A player that we're not even talking about right now is going to be crucial to fantasy football success this year. A team, it isn't even a trendy team to be good out of the 32 teams. No one's picking to certainly improve is going to improve massively. It is such uncertainty and when I finally arrived at, in terms of my relatively successful, I did not want to play fantasy football in the league with Adam. Rob Collie (00:55:46): I'm not going to be that guy, and home leagues level of success, home and work league type of success is that it's just going in and just saying, "Look, my entire strategy is going to be based around the idea, the admission that none of us know anything." And you actually behave differently when you accept that. You make different decisions. It's not just about expectations. Rob Collie (00:56:12): You decide differently, and one of the things that I've seen that I've been doing for a long time and it's been good to me when I start listening now, like this year, I've really been listening to a lot of podcasts about fantasy football, it's the first year I've ever done this. And it's so amazing to see how widespread this notion is, that we really primarily only care about the best possible outcomes. When you're playing a zero sum game where you're one out of 12 or one out of 14, in order to win, you need exceptional things to happen in your favor. Rob Collie (00:56:42): And you need to maximize your chances of getting those exceptional things. To use a baseball metaphor, singles and doubles aren't going to win you the title. You need to be hitting triples and home runs and the nature of football, plus, I find this emotionally satisfying, it's like prospecting for gold. I'm not going to be drafting players or picking players who I know to be predictably mediocre. I'm going to be drafting and selecting for best possible range of outcomes and if they turn out to not be exceptional then I'm going to get rid of them. Rob Collie (00:57:12): This isn't genius, this isn't some new strategy. I hope that people at our company who are going to be competing in fantasy football against me aren't listening. but some of them probably are. I used to think that I might actually be able to pick players. You can't predict what's going to happen in an NFL season. It's crazy. And I love that. Adam Harstad (00:57:29): Real quick, I want to push back. You were saying you put up the list of rankings and then you were saying, "Ha-ha, they won't even last a week." Actually, empirically I found pretty consistently looking at it over the last 10 years that preseason ADP average draft position, the order that players were drafted in before the season better predicts rest of year performance than current year to date performance does until about week four. And that's when they retract parody. Adam Harstad (00:57:56): They will last a week. They'll last approximately four weeks, is one way to look at it. Or I guess another way to look at it is if you just drafted off of last year's order of finish minus guys who retired or who are clearly injured, things like that, that will be more predictive of year to date production up until about week three. My stance is everything we do in the off season, all of the analysis, all of the... Everything buys us one more week before we're wrong. If we did none of that, we could make it till week three before your due date was more powerful. Adam Harstad (00:58:28): But by doing all of that, it buys us one extra week before we're wrong. Rob Collie (00:58:33): That's amazing. Adam Harstad (00:58:34): In that sense, it does get you one extra week. Rob Collie (00:58:36): I just got snared on my own flare for hyperbole. Adam Harstad (00:58:42): I just figured you would like to put some numbers to it. Rob Collie (00:58:43): And I wouldn’t have been able to do that. Adam Harstad (00:58:45): And now you can. Rob Collie (00:58:46): Now we absolutely can. I loved that your arc so far in life goes from some sort of genius to the niche where I know nothing. There's something truly delicious about that and evolutionary. Adam Harstad (00:59:02): The fool thinks himself wise and the wise man knows he's a pool. Rob Collie (00:59:04): Exactly. And you can never pronounce yourself cured of this, of this disease because the moment you do now, you're back to the trap. You can never pronounce yourself healthy in this regard. It requires vigilance, constant vigilance and constant growth. But I like to think that I've had a similar arc, I've been on a similar arc from the know it all to know nothing and becoming comfortable with that. Did you watch Ted Lasso? Adam Harstad (00:59:31): I've seen the first season, yeah. Rob Collie (00:59:32): The speech he gives about be curious in the bar right before he kicks the guy's ass in darts? Adam Harstad (00:59:38): Yeah. Rob Collie (00:59:38): I found that to be one of my top 10 favorite moments of anything I've seen on any screen, was that sequence. Be curious. Adam Harstad (00:59:46): The cool thing about being wrong, it sucks being wrong, there's that whole ego threat where we identify ourselves with our positions and it's hard to get past that, even somebody like me, who's had so much experience being wrong. There's that ego threat that I don't want to be wrong, but the cool thing about being wrong, it's the first step to stopping, being wrong. You can't stop being wrong until you realize you're wrong in the first place. Rob Collie (01:00:09): It is a very difficult thing to come to terms with. I'm much better at it than I used to be. That's the only thing I'm comfortable saying. Again, I got to be careful. You can't be the one that says, "Yeah, I used to be like that, but I'm not like that anymore." "Yes you are." Adam Harstad (01:00:26): It's like modesty. It's impossible to declare yourself modest. That's right. I am probably the most modest person you've had on this show. If I'm being honest here. Overwhelming modesty. Rob Collie (01:00:38): If I say modesty is not my strong suit, am I being modest? Adam Harstad (01:00:42): I can self-deprecate better than anyone. I'm the most self-deprecating person you'll ever meet. Rob Collie (01:00:50): You know this from an emotional standpoint even from past years. After one week of play, you still would want to draft very differently based on what you saw. Adam Harstad (01:01:00): I think that's a trap though. Rob Collie (01:01:01): I agree. Adam Harstad (01:01:01): Yes. Emotionally people way overweight and their priors aren't weighted enough. They're reacting too much to new evidence and that's why I write that piece every year around week four and I say, "It might surprise you to know that pre-season ADP at this point still has every bit as much predictive power." Now, obviously some blend between the two, a mix of pre-season ADP plus year to date results will outperform either alone, but it's really a lesson about not discarding what we thought we knew just because somebody makes a huge, impressive catch on Monday night football. Rob Collie (01:01:32): Well, let me see if I can make a subtle, not a point, but a question. There are situations where we can talk about this is our best possible tool. It's 40% better than chance at doing something. You'd still be better off betting, if you could bet the field, if you could bet against the predictions, you'd still do better. There's a weird, subtle distinction there between best possible predictive power, which is factual, empirical and at the same time betting that it was wrong. The ha-ha still has some less than truth to it. Adam Harstad (01:02:08): Yeah. I'd say my entire niche in the industry and it's a weird niche to occupy, but I think my entire niche is just I'm the guy who knows nothing. I just aggressively embrace that uncertainty and rather than trying to know more things, I feel like I try to expound the limits of the things that I don't know. It really does change the way that you play and you manage your teams. And one example, one of my favorite things that I've done is about six years back, I looked at how players age and the popular conception of how players age is something called an age curve. Adam Harstad (01:02:44): You enter, you improve for three or four years, you hit your peak, you maintain your peak for however many years, depends on your position. And then you decline and then you're out of the league. And if you plot all those points together, it looks like a curve. And there's a lot of ways to drive this curve. If you average performance at every player age, it'll make this beautiful curve. And the age curve is an example of thinking that you know something. If you look at a player, the age curve will say, this player is going to do this this year, this next year, this next year, this the year after that. Adam Harstad (01:03:12): He's going to be out of the league at this age. It represents certainty and I never liked age curves because if you actually look at NFL careers, very, very rarely do they take on a curve shape. There's a lot of fluctuating, a lot of up and down. There's very few players who I would look at their career and say, a curve really describes what's happening here. And so, I gathered 30 years worth of data about year to year fantasy football value. And I said, "Let's try to make a mortality table out of this instead." Adam Harstad (01:03:42): If you're familiar with life insurance, in life insurance, they're basically making a bet for a life insurer that you will survive to this age. You'll survive to that age, but it's not a curve. It's not like you're 80% alive at age 64. You're still either a hundred percent alive or you're a hundred percent dead. It's a very binary state. They do tables that look at it probabilistically. And I looked at NFL player careers. And so, one thing is if careers are curve shaped, you would expect a player's last relevant season to be worse than his second to last relevant season, on average. Adam Harstad (01:04:13): But I looked at the top hundred wide receivers and running backs of the last 30 years and it was about 50, 50. 50% of them did better in their last season, 50% of them did worse than their last season before they just fell out of relevance completely. And so, I did, I modeled and I found that I think the mortality table is actually a pretty good fit for what's happening is players are basically reaching their true level of play. They ascend, they learn the position and then they reach their true level of play. Adam Harstad (01:04:39): And once they reach that level of play, they pretty much maintain that level of play. They'll have up years, they'll have down years, but they're fluctuating around that true level of play until at some point and usually without any warning whatsoever, they just fall off completely. They're great. They're great. They're great. They're done. And when you start thinking about aging like that, this wide receiver is 31, so I expect him to be a little bit worse than he was last year, and it's more, this wide receiver is 31, so I expect him probably to be about as good as last year or also maybe he will be completely useless. Rob Collie (01:05:11): Love it. Adam Harstad (01:05:12): And the way you manage your risks is very different. When an age curve would tell you that a 28-year-old wide receiver is a very safe bet. He's still on the upslope of his curve. He's going to get better or he's going to maintain, you have years before you have to worry about him disappearing. And you look at Dez Bryant who is an All-Pro wide receiver, never did anything after age 28. He was done and an age curve would never predict that. Whereas a mortality table would say, it's unlikely, but this is well within the range of possibilities. Adam Harstad (01:05:40): I played dynasty leagues where you keep your players from year to year, basically indefinitely, and it really makes you hyper aware of the risks involved. You never enter a season thinking that you're set at any position because you look and you're like, "Well, there's a 20% chance that this guy's just done. There's a 10% chance that this guy is just done." You add up all of those chances across everyone. And a lot of guys on my roster are never going to have another good season again. Adam Harstad (01:06:06): And on the flip side, sometimes you're more likely to take a chance on old players because well, everybody thinks he's in the decline. Julio Jones would be a great example this year. Everybody thinks he's going to be worse this year, but maybe he's not. Maybe he's just exactly as good as he has been for another five years. That's in the range of possibilities. It's unlikely, but it's not fundamentally unforeseeable outcome. Rob Collie (01:06:31): This tickles two ideas from daily life, whatever. One is, I have a joke, which is the average has a population of zero. When you were describing the curves, this is the aggregate average of everything. All the players that have ever lived. That curve never describes any one player. Adam Harstad (01:06:52): It's called the ecological fallacy, if you want a name for it. Rob Collie (01:06:58): Ecological fallacy. I think you'll appreciate this. From the business side for a very, very, very long time, honestly, even to this day, really, this still persists a lot of companies, they have their top number. They have their top level number for whatever metric it is, profit margin or customer retention or whatever. They know what their all up number for the entire company is, but they lack the ability to decompose it into the various segments of their business. If they've got 50 salespeople out there, what's the average win rate for a deal? Rob Collie (01:07:28): Now, let's say it's 25%. But if you go look at the 50 salespeople, none of them are at 25%. You've got some that are down at 3%, and then you've got a handful of outliers at 45% and you're not learning anything. There's no way to improve. One of our core philosophies is you need to know your top level number. You absolutely do. You need to know what your final score is, but the game is improvement. It's not knowing. Rob Collie (01:07:54): Knowing is a necessary step for improvement and you cannot improve unless you're able to effortlessly subsegment and see what's really going on to focus your attention in the right places. There's different prescriptions for different places. And so, when you're talking about this curve versus mortality risk, I was lighting up like, "Yeah." And the other thing that I've been wrestling with and really just torturing some of my friends with for a long time is this notion of single trial experiments and probabilistic models applied to single trials. Rob Collie (01:08:27): Julio Jones isn't going to have an 80% season probably. He's not 80% relevant or 80% still in the league. He's either going to have a good season probably or vanish from relevance. Probabilistic models of who wins the election. I know that's our best tool for predicting who's going to win an election, but there's something fundamentally silly about saying that someone has an 85% chance to win the election. They're not going to 85% win ever. They're going to win or they're going to lose. Rob Collie (01:08:56): And so, we need to come to terms with that, so when it's wrong, when something like that is wrong, it's more wrong than 15% wrong, in my opinion. Anyway, we don't need to pick that fight. I've done that a million times, a million different people. Adam Harstad (01:09:11): Nassim Taleb versus Nate Silver debate. Rob Collie (01:09:13): Yeah. I Think I'm on the Nassim Taleb side of that debate. Adam Harstad (01:09:16): I tend to be more with Nate Silver, where on any given prediction, you can't know if it was a good or bad prediction, but over a large enough sample, you can test your calibration. And if the things that you say will happen, 15% of the time are happening 15% of the time, I don't think it says anything about the predictions themselves, but I think it says something about you as a predictor being fairly well calibrated. Rob Collie (01:09:38): Sure, but of course, all that matters to me is its accuracy and of course that includes both the predictor and the methodology. Adam Harstad (01:09:45): Well, here's an interesting thought experiment. Let's say we have two different models. We'll go back to Ja'Marr Chase, rookie wide receiver for the Bengals, and we have two different models and these models are both extremely well calibrated. And you feed them different inputs and they will tell you what percent chance a player has of becoming a Hall of Famer. And we've tested these models, and we know, let's just say by faith that these models we're going to stipulate that they're extraordinarily well calibrated. Adam Harstad (01:10:11): And one model says that Ja'Marr Chase has a 15% chance of becoming a Hall of Fame wide receiver, and another model that he has a 40% chance of becoming a Hall of Fame wide receiver, because it's looking at different inputs and those inputs predict different chances. You know what are Ja'Marr Chase's true chances of becoming a Hall of Fame wide receiver? Is it 15%? Is it 40%? Is it somewhere between the two? If he does become, how right or wrong was each prediction? If he doesn't become, how right or wrong was each prediction? Adam Harstad (01:10:40): Two predictions, those can both be right. It seems fundamentally in conflict that one person says he has a 15% chance, one person says he has a 40% chance and both can be right at the same time, and yet it is. We're stipulating upfront that both are right. And I think when you grapple with data and especially predictive data, you need to develop a certain comfort level with that ambiguity, with we're just never going to know. We never will know. And there's no way we possibly could know, because as you say, it's a single run trial. We can't rerun it. Adam Harstad (01:11:12): But if we're aggressively testing our calibration and we're working hard and making sure that the things that we're saying are correlating with reality to some extent or another, it's almost like you're not really having faith in the predictor so much as you're having faith in the process to produce results that will have meaning and add value. Rob Collie (01:11:32): The most sophisticated version of my objection to all of this is actually, and not to what you were saying. I completely agree with the things you were saying. The most refined version of my objection to Nate Silver is that I think the domain he's applying these methods to is exactly the wrong one to give these methods a positive reputation, because here's the thing, on the eve of the election, the universe, this is can be a little weird. We're going to have to get a little weird to explain it, but the universe actually knows who's going to win. Rob Collie (01:12:03): There's no chance that it's going to swing 85, 15% tomorrow. That's not going to happen. We already know. The universe knows. Whereas in the case of Ja'Marr Chase, I can make this fuzzy argument that even the universe doesn't know if Ja'Marr Chase is going to be successful. Adam Harstad (01:12:18): I don't know if I took that argument. I have a thread on this on Twitter. That's saying basically what you’re saying. If I flip a coin and I say, "What are the odds that this coin comes up heads?" First, you want to know if the coin is biased, assume it's a fair coin. It comes up heads 50% of the time. I flip a coin. I say, "What are the odds this comes up heads?" You're going to tell me 50-50. Adam Harstad (01:12:36): If I were flipping this coin in front of some alien superintelligence who was so innately good at physics that he could divine from the starting point and the forces involved and wind resistance, he could tell with absolute certainty, which side was going to come up. He'd either say a hundred percent heads or 0% tails. And again, both of you are right. You say 50-50, you're correct. The alien says a hundred percent or 0%. If he's well calibrated, he's also correct. Fundamentally odds are not measuring anything intrinsic to the event itself. Adam Harstad (01:13:06): Odds are a measure of our ignorance surrounding the event itself. And people will get different odds for different things because they have different levels of ignorance. If I hold up a card from a deck facing me and I say, "What are the chances this card is an ace?" You're going to give me one number. If I then show you 13 cards from the deck and say, "What are the chances these cards are an ace?" You're going to give me a different number. The card itself hasn't changed. Rob Collie (01:13:29): I know. Adam Harstad (01:13:30): The odds are really meaningless to the card itself. They're just a measure of your level of knowledge and what you know. The odds are saying more about you than they're saying about the card. Rob Collie (01:13:39): Well, let me push back a little bit. Let's take your alien intelligence coin flip example. And instead, have it be a billiard ball on a frictionless surface or damn near frictionless surface, so that we could make the billiard ball bounce off of the rails 40 or 50 times. Once you get that number of collisions, maybe 50 isn't sufficient, maybe it takes a hundred. Rob Collie (01:14:01): Once you get the number of collisions high enough, even the alien intelligence won't be able to predict where it ends up at the end of a hundred bounces because even the quantum uncertainty and the vibration of the ball magnifies to great enough difference that you don't end up in the same place. Adam Harstad (01:14:15): Well, but the quantum uncertainty, that's what I'm saying. Again, the odds are a measure of our own uncertainty. Rob Collie (01:14:20): Completely. Adam Harstad (01:14:20): So if you increase the alien's uncertainty, you're going to decrease the odds [inaudible 01:14:24]. Rob Collie (01:14:24): Okay. But I'm saying, the distinction between the two. No amount of tomorrow uncertainty is going to change the outcome of the election. All that uncertainty balances itself out, large numbers. Whoever wins the election was going to win the election in a million sub universes or a billion sub... Whatever. They're going to win on all of them. Whereas Ja'Marr Chase, I think even the universe with perfect knowledge, you don't actually know. There's too many branching points in the equation of whether he turns into... Rob Collie (01:14:52): And this is why I think intuitively the masses have a point when they make fun of Nate Silver getting it wrong. Whereas they would be more accepting of the Ja'Marr Chase Hall of Fame percentage. We understand it's uncertain, but they get that there was a more wrongness about the election prediction than there is about the Ja'Marr Chase prediction. Adam Harstad (01:15:15): Yeah. I think the famous quote is, "All models are wrong, some are useful." And that's the question, any model. What you're getting at, I think is, there's something called epistemic uncertainty and something called aleatory uncertainty. I know you like learning new- Rob Collie (01:15:31): [crosstalk 01:15:31]. Oh my God. Write these down, Luke, I need to... Adam Harstad (01:15:35): Epistemic uncertainty is things that are theoretically knowable, we just don't know them. And I think you're suggesting that an election outcome, the day before that's epistemic uncertainty. This is fundamentally knowable. We just don't know it. And the odds that we're assigning are representing that epistemic uncertainty and aleatory uncertainty is irreducible uncertainty. It's even if we knew absolutely everything, this is fundamentally unknowable, it's irreducible. Adam Harstad (01:16:03): And it seems to me that you're suggesting that probabilistic forecasts in the case of aleatory uncertainty are okay. Something like Ja'Marr Chase is going to make the Hall of Fame, he won't make the Hall of Fame. Here's this chance. Whereas in the realm of epistemic uncertainty, you're saying there's just something hinky about them. There's something fundamentally weird uncomfortable. Rob Collie (01:16:23): It's not that they're not okay. It's just that they're a little less okay. Adam Harstad (01:16:27): It's fundamentally different. Rob Collie (01:16:28): An 85% number about something that is only going to be zero or a hundred, and I believe we'll be the same zero or a hundred, no matter what happens in between now and then, there's something very funky about even bothering to put a number. It's like the illusion of confidence. It's the illusion of precision. 85.6%. Not 0.5%, 0.6%, Nate? It's like what you were saying earlier like, when you're deep into analytics, you also have to maintain that other wisdom. Rob Collie (01:16:59): I just think Nate Silver in particular, this forecasting, the result of a national election, it's just the worst possible poster child for the analytics industry. We're drawn to it. We look at it like crazy, but just the worst place to have our reputations as professionals displayed. Adam Harstad (01:17:16): Yeah. It's funny. Footballguys, we obviously the people who write for Footballguys who are big into fantasy football we're pretty fluent with data and probabilities because what you're saying about your BVBA, that's basically data and probabilities. That's what fantasy football is. And we had a really interesting discussion after the 2016 election where some people were saying Nate Silver was very, very wrong because he said there was only a 36% chance. And other people were saying everybody else had the chances at 1% to 10%. Adam Harstad (01:17:45): Nate Silver was the only one out there saying there's a very real chance. 36% might not sound that much, but it's about the odds of a kicker missing a 48-yard field goal. If your team is lining up to kick a go ahead field goal from 48 yards with time expiring, how comfortable are you? That's what 36% feels like. It happens a lot, 36% of the time in fact. It's interesting that even in a place that's so comfortable dealing with risk and probability, there's still that fundamental divide and I think it really boils down to the appropriateness of putting numbers on epistemic versus aleatory uncertainty. Rob Collie (01:18:20): And I was with you for a long time that there's this over precision, but Phil Tetlock, I don't know if you're familiar with him, works on a project called The Good Judgment Project. And he wrote a book called Super Forecasting. And basically the CIA had this forecasting competition. They have a hundred questions. Here is this Middle Eastern country. What are the chances that they're going to suffer a coup in the next seven years? And Tetlock basically his approach to it was this wisdom of the crowd approach. A modified wisdom of the crowd. Rob Collie (01:18:49): Is he had a bunch of people making predictions, and then over time he saw that some people were more consistently right and so he started weighting their predictions more and more. And eventually he came up with this group of people who were definitely out predicting everybody else, usually with no specific domain expertise. And he named them super forecasters and then he's done a lot of studies about what makes these people so good at predicting basically one off events. And he found that when you look at the precision, the precision is meaningful. Rob Collie (01:19:17): If a super forecaster says there's an 83% chance that there's going to be a coup in this country versus an 80% chance, instead of just rounding to the nearest 10, he finds that that 3% is meaningful. That ignoring that 3% makes the aggregation worse, makes the outcomes less predictive. And so, I would say that for the most part, I'm skeptical of that appearance of certainty and that appearance of precision, just because most people are not super forecasters, but I have become more open to the possibility that actually no, that 0.3% might be doing some real work there. Adam Harstad (01:19:50): Yeah. I think the distinction again, when I'm not being just bombastic about it, when I'm really clear about it. My only objection here really is the reputational risk inherent. Rob Collie (01:20:03): Which is fair. Adam Harstad (01:20:04): I do want, if we're going to be making predictions, of course I want the methodology to be rigorous, if you're not artificial about it, the methodology that produces more precise numbers probably has more internal checks in the first place. It probably is the process that you use to arrive at that number was probably a superior process to the one that arrives at the round number. I'm okay with the methodology part of it. It's just that when we put it out in the public, it's the old nerds versus jerks argument all over again. Adam Harstad (01:20:34): But now we're in our middle age and the whole notion of using data to do things gets cheapened every time we put an 85% number on an election, and then it's either a much closer than expected or it goes the other way. I just wish again, for reputational reasons only, it's the rear view mirror, not the forward only allowed three predictions for an election. This person wins, that person wins, too close to call. Tom Larock (01:21:00): I just want to say, Rob, I'm not going to disagree with anything you've said, but what you're missing is the fact that 86% of Americans don't understand what the P value really means. Your talk about, this is a bad thing to put in front of people, but the real thing is, is that Nate Silver goes out and he describes things as if people understand what these concepts are and they really don't. Rob Collie (01:21:29): I'm saying two things. One is that you go to war with the army you've got right. The population that you've got. And so, they don't understand P value is great. But the other thing I'm saying is that there's something also fundamentally meaningless about his P value. It's because of this different aleatory versus pistemological, was that it? Tom Larock (01:21:46): Episcopalian, I thought. Rob Collie (01:21:48): [crosstalk 01:21:48]. Tom Larock (01:21:48): Episcopalian Rob Collie (01:21:52): Epistemic. Adam Harstad (01:21:54): If I can have one rebuttal. Rob Collie (01:21:56): Of course, we'll give you the last word. Adam Harstad (01:21:58): I'm very, very respectful of arguments that it's not enough to be right, it's are you advancing the cause? I think trying to think tactically, that's a very important skill to have, but I would also say, I don't think that there's been anybody in America who has done more for the understanding and acceptance of analytics, than Nate Silver over the last decade. I think the public is much more open to conversant in the language of analytics and I think that stems primarily to Nate Silver in FiveThirtyEight in his election forecasting. Adam Harstad (01:22:32): Maybe the 2016 election was a setback, but I think on the whole, he's been a very positive, because I see it in fantasy football too. Just the acceptance of alternate data sources. 15 years ago, you mentioned Football Outsiders and people dismiss you out of hand. Nowadays you get random people off the street who are citing DVOA and EPA per play to you. And I think Nate Silver was a big part of that revolution. Rob Collie (01:22:56): I agree with you. I think that's true. Can I not disagree, but adjust it a little bit? Let's bring back the average has a population of zero. I think he's probably reeled in some percentage of the population to the idea of analytics while pushing another percentage further away. Adam Harstad (01:23:15): Maybe. Rob Collie (01:23:15): But then in terms of real numbers, we're seeing, I totally agree with you. The analytics has benefited as a field from the exposure he's brought to it, for sure. But even if that were, and again, we're just making up numbers at this point, even if that were 20% of the world that was now more receptive as a result, that would be massive. But I see a lot of ha-ha out in the world and I just don't want the methodology that he uses to be unfairly judged by the domain in which we find it most entertaining. Rob Collie (01:23:48): And I do think that the reason why he's so famous for the politics stuff is more for the entertainment reason, which is not the right reason. I don't have any objection to the guy or his methods or anything like that. It really is just like, "Ugh, I wish we could be fascinated with something else." Adam Harstad (01:24:06): Maybe. Rob Collie (01:24:06): Adam, man, a thank you is appropriate because I've enjoyed the hell out of this. Adam Harstad (01:24:10): I did too. It was a lot of fun. Announcer (01:24:11): Thanks for listening to the Raw Data By P3 Adaptive podcast. Let the experts at P3 Adaptive help your business. Just go to p3adaptive.com. Have a data day.