The Reality of AI Economics With Paul Kedrosky

00:00:02
Speaker 1: Also media.

00:00:05
Speaker 2: Hello and welcome to Better Offline. I'm your host ed Zeitron. Download a T shirt and subscribe to the newsletter that's where you get your words. But today you're here for the noises. And joining me today is the wonderful economist Paul Kadrowski.

00:00:29
Speaker 3: Paul, good to have you on.

00:00:31
Speaker 1: Hey had good to be here.

00:00:32
Speaker 3: So your recent work.

00:00:33
Speaker 2: On AI, particularly the data center on economic side, it's been great. One thing I really want to talk to you about is what actual economic effects have you've seen from AI, because it feels hard to get specific sometimes if you know what I mean.

00:00:48
Speaker 1: Yeah, so it's pretty easy to find that. I mean, there's the old joke, which was the early days of the technology industry that you could find technology everywhere except for in the productivity data. And so that sort of applies right here as well. So the answer you'll get there's two answers. One is that it's too soon to tell, which is fine, but it's a little bit of a sop, right, So you'll get that, and then you get the second answer is which I already see it. It's in. And then someone will add hoc Terry picks some data and say there goes Ai right there, And the answer, of course, is is that it's nowhere to be seen yet in any really meaningful productivity data. Anywhere you have some output metrics like for example, you know this incredible number of commits you'll see on GitHub using agentic code. But is that productivity? I think it's largely masturbatory. I don't think it's actually, I always productivity. And so most of what got me interested was because you see it all on the other side of things. So from an economic standpoint, you actually see it in the economic data because of how dominants become in a couple of statistics, like for example, it's share of non residential fixed investments, which is at levels we last saw with the railroad build out or with rural electrification.

00:01:55
Speaker 2: And there can you be specific, Well, non residential investments may just so I'll get you.

00:02:01
Speaker 1: So atoms that aren't in houses. So that's the short answer. So so factories, manufacturing highways, Uh, you know, fixed equipment you're putting in your in your in anything that you're building for an economic purpose, it isn't residential, Okay, So that usually is a fairly broad and diversified category. So you've got people building out. You know, if the current administration had its way, that statistic would currently be dominated by fill in the blank manufacturing, right, because the attempt is to onshore manufacturing, so that if policy was working in that regard, you'd be seeing the dominant chunk of that being the onshoring of manufacturing. As it came back to this, to this blessed country. And so it's not the largest chunk of non residential fixed investment currently, which is data centers, which is a basket of things which it's made up predominantly of GPUs, but also the build out of the of the of the facilities, HVAC, heating, ventilating, air conditioning, cooling, all those kinds of things. So what got me interested originally was I couldn't see AI data anywhere except for him these categories of fixed investment. And then even more startling to me, which apparently I was the only one initially startled, then I startled other people, was the idea that it was the largest share of US GDP growth for three or four quarters last year, so that in the absence of this non residential fixed investment wonder drug we call data centers, US would have been in recession in the first quarter and the fourth quarter, neutral in the second, and mildly positive in the third. So it was the economic story of twenty twenty five. And so the reason why this is important, and the analogy I make all the time is my dog barks when the mailman comes to the house, and the dog keeps barking and the mailman goes away. Right, the dog thinks he did it.

00:03:51
Speaker 3: He didn't do it right.

00:03:52
Speaker 1: He has a messed up He has a messed up model of causality is what he has right? The dog causality imply is that my barking made the male men go away. No, the mail man goes away every time. It's got other things to do. Right. So the same thing is true with respect to fixed investment. If you don't realize that the largest share of fixed investment and the thing that's driving us GDP growth is this wonder drug called data centers, you're the dog barking at the mailman and the mailman goes away. Anyway, you don't actually understand the things that are driving economic growth. So there's the long answer is that's where you see data centers in economic data, and it's in this very strange place that was largely being missed for the longest time.

00:04:33
Speaker 2: So with that number, that includes all of Nvidia's sales all told, or just the ones in going to American clients or is it just all of their sales?

00:04:44
Speaker 1: No, So you parcel it out obviously so by geography. So what matters immensely what's happening in the US? No, Granted, the predominant share of in videos sales, just like the largest sales of most of you know, transformers and everything, are all happening in the US right as this build out happens. Some like seventy to eighty percent of the global data center build out is happening in the US, and most of that is happening in northern Virginia, Texas. So it's largely a US phenomenon in the first place. But nevertheless, you have to parcel out the pieces of Burbrialley.

00:05:12
Speaker 3: Yeah, I just meant it more.

00:05:13
Speaker 2: Is when in video makes a doll that does that count into this? Because yes, but see that's the thing that feels like something that is just kind of almost like a load bearing chip, a load bearing GPR. Like if these sales go down, as bad for everyone.

00:05:31
Speaker 1: Right and there's been some tremendous piece the Wall Street journally at a piece last week, I think, and I said some salina and scendiary things in it, but it was about how in Vidia kind of sits at the center of this, like you know, Don Carleoni and sitting with everything happening and everyone coming to the table, and they're this you know, mafia Don, who is investing in things, right, So they play the role of investor, they play the role of acquirer, they play the role of you know, vendor. So they have this this incredible hub role. So each dollar that they're putting out there, in a sense, is vastly more important because in a sense they are the load bearing beam in the middle of all of this. They are for now and it's changing rapidly, but nevertheless they are for now.

00:06:09
Speaker 2: The thing is it that just feels very unstable to me because in video, from what I've worked out for them to keep growing at their current right, they're going to be selling one hundred and twenty billion dollars of GPUs by in a year and then in like Q two f y twenty eight, I think it will be next year. Yeah, And that's just yeah, that feels impractical, like just on a on a like an economic level, for any country or anyone investing.

00:06:34
Speaker 1: I have a tendency to make that argument, and I hate when I do it because and this is so we both have fault for this is this is Paul's argument from personal incredulity. I don't think it can happen, Therefore it can happen. So let's take them at phase So let's take them at phase value. Let's say they're right right, say this is what's going to happen. You have to look at the dynamics. And they conceded this at GtC their most recent conference. The dynamics are changing quickly in the marketplace. That's driving two things, sales growth in margins. So Nvidia has these anomalous GPU margins in excess of seventy percent gross margins, which are ridiculous.

00:07:07
Speaker 3: And that's the serious that's right exactly.

00:07:10
Speaker 1: And so so their margins are very high. But they're in the middle of this transition, this admitted transition from what's euphemistically called training, which is kind of a misnomer, to this thing that's called inference, which is probably more accurate. So from training models to answering prompts right, and the margins on inference are going to change dramatically because the things that gave them and this is one of these Silicon Valley silly words, but gave them a moat, the things that gave them a moat in the world of training are far less important in the world of inference. And so you're saying this proliferation of new chip companies coming to market incumbents with new products, so they're facing much more competition in a world of inference. So even if you grant that the market's going to grow as large as it once did, which is whatever, it's not going to all go to them. They're not in the same position to accrue all that benefit that they did almost accidentally in the world of training, which is really an important point as well.

00:08:04
Speaker 2: See that's the thing. I'm also questioning the demand, and I also question whether they're done with training because training you correctly said, it's a misnomer because that can mean everything from a big pre training run to the post training that's necessary to make these things work. And it's kind of confusing at the moment because like last year, they were saying it's all inference all the time. This year they're kind of talking about open Claw. Kind of feels like they're a bit lost, which I mean is kind of the AI industry at large.

00:08:32
Speaker 1: It's just it's so the way I look at it is I always whenever someone says in video, I say Saudi Arabia, and when they say tokens, I say humpies. Right, So their goal is to get more people. Their goal is to get more purchase for people purchasing humbies because it's good for them because it consumes the thing they, however, indirectly produce, which is to say, these things called tokens, not the crick Right. So if you think about it in the terms and translate Jensen's you know, GtC talk and most of the things he says, it doesn't read that differently from a random industrial minister in Saudi Arabia saying, you know this oil stuff, If you guys, just back away from the evs before it hurts, you get out there in the humvies so you're safe on the freeways. It can kind of feel equivalent.

00:09:19
Speaker 2: And did that actually was that something that happened? I was that like two thousand and eight. I remember there was like the stories about empty like parking lots full of just unsold cause yeah yeah right here right.

00:09:31
Speaker 1: Yeah, no, no, no exactly so So so I think you have to translate a lot of Jensen speak into this kind of idea that there is this new commodity emerging, no different than oil, no different than I don't know, copper or whatever else, and this new token is this or this new commodity is this thing called tokens, and he's doing what he can as a diligent ambassador for the for this commodity called tokens to make sure people use as much as possible, like, for example, endorsing this completely half hour, wild eyed thing called open claw, which is a ridiculous idea to suggest that people should be using this in their house. It's like I should have you know, my own sort of you know home, uh, nuclear reactor. It's it's it's a ridiculously dangerous technology for most normals to be using.

00:10:13
Speaker 2: Yeah, I just I also feel like it's a signed there, a little washed when it's you've got a three D AI generated picture of Jensen Huang, the CEO of a company with a multi trillion dollar market cap, with crab claws like the indignity of it. He wears seven dollars jackets. Man, one man should have a little more swag than claws. It's just tremendous jackets though, oh trum, they are trump at the men'swag guy on a few years amazing jackets the man does.

00:10:40
Speaker 3: Here's a good tailor as well. But any who, So.

00:10:44
Speaker 1: Yeah, so I think that's that's a really important point you make though that I think you have to when you're The read through on open Claw isn't just some confusion about where the market's going, but the read through is the promotion in almost an industrial affairs level of this commodity called token to how can I get people to use more of this? So that's why you hear song and dance acts about open Claw, these incredible hyperbole about cloud code and how cloud code is going to rapidly migrate from the world of software into all of white collar work, which is agam and error. It's not to say cloud code isn't a really interesting and important piece of technology, but people are very misguided about how these technologies are going to move or can move, or will move from a world of software, which is wildly anomalous in terms of both the amount of tokens it produces, but also in terms of whether you can leave it alone. So think about it by the way I sometimes think about it is in terms of the idea of like a ground truth. I can look at my software and I put create some code and change it and it breaks. In AI terms, that's a really tight gradient descent, meaning that obviously I've just learned something really quickly. Changing this to this doesn't work in most of white collar work. That's not true. What matters is I create a PowerPoint presentation, does my boss like it? Tell me the gradient descent there? There's no gradient to set. It's very subject if it's almost an aesthetic answer. There are parts of white collar work where that's not true. But much of white color work doesn't have the same characteristics as software. So if you want to project the kind of growth that people like Jensen are projecting, you need to believe that these harnesses cloude code, codex blah blah blah.

00:12:18
Speaker 3: Yeah light that you've used with the models, Yeah.

00:12:21
Speaker 1: Yeah, yeah. You have to believe that, like the Veloci raptors in Jurassic Park, that they can escape containment, that they're going to escape containment, and they're going to get out of this corral that we call software and they're going to be everywhere. And not only are they going to be everywhere, which is happening to a degree, they will act in the same way, which is to say, they will produce huge amounts of code for tiny or huge amounts of output for tiny input. And they can be left alone because there is this tight gradient descent that tells them whether what they're doing is working or not.

00:12:48
Speaker 3: That's things they don't know. If that like, they don't know anything, So you don't guarantee that.

00:12:53
Speaker 1: Because there's no ground truth, right, So this gradia descent doesn't work. It doesn't work in almost any domain outside of software. So the weird thing is is we've actually started off in the nearly perfect domain to give a completely unrepresentative example of what the future looks like by start off in coding, which is why coding or the coders and developers are some of the biggest sort of flag waving ambassadors for what's going on. It's like, just wait till this shows up in you know, I don't know it. Pick your domain and why client work, and the reality is those domains are very different.

00:13:30
Speaker 3: It is strange as well. Like the.

00:13:33
Speaker 2: You get very few people who are just like, yeah, I kind of like this. It's either the pure hatred, which I name my listeners is definitely or there's just this cult like thing around it. I've never I've been around tech for a while. You've probably been around it longer. Yeah, I've never seen anything like this. And I've been on web forums or games consoles.

00:13:53
Speaker 3: I've been on.

00:13:54
Speaker 2: I've been on I've used to because I'm a strange person, read bodybuilding forums and BMW forums, and the arguments on there were the same, like if you don't drive an M three, I will run you over with mine. Then that kind of thing. I've never seen anything like this happen. Like I maybe in stocks, yeah, get.

00:14:12
Speaker 1: In stocks to a limited degree. Right, So there's these two overlapping phenomena going on. I'm convinced one this is a tribal signifier. I show what tribe I'm in by by being as wild eyed as possible about my support for this shows I'm in the in group or the outgroup. And you see that, right, this sort of tribal signifier stuff. And the other one is and this is the one that I find most fascinating is this kind of frantic quest to believe that if we work hard enough and fast enough, that I'll never have to work again. And you hear this from the people in the Acceleration on, the Acceleration camp, the Singularity camp, that when you penetrate deeply enough, it's really just I think if I'm kind of creating, think of me as as a bomber and I've got a vest, I've got a vest attached with all these explosives, and I'm threatened to walk into your economy and blow it all up, right, and just try and stop me, because there's hundreds of us doing this, and we're all going to come and we're all going to blow up your economy, and what are you gonna do about it?

00:15:06
Speaker 2: Right?

00:15:07
Speaker 1: And so that's what I think a lot of this is because they feel like if I do that, then the government has no option but tends to do some kind of broad policy of support where diet I can then go out and you know, make figurines all day or whatever it is I want to do.

00:15:19
Speaker 2: Yeah. But I will also say that the people who are most excited about a I don't seem to have other hobbies.

00:15:26
Speaker 1: That's always the thing of fun. It's totally true.

00:15:28
Speaker 3: What do you plan to do with spare time?

00:15:30
Speaker 2: I'm learning piano right now, I'm learning to code for fun. And it's like I look at these people and they're.

00:15:35
Speaker 3: Like, what do you do? I run seventeen sub agents an hour.

00:15:38
Speaker 2: If these things, if these things don't create my special project that I'll never show you, I will die.

00:15:43
Speaker 1: It's like to gray out the power in my neighborhood by running as many subprocesses as possible. Yeah, there great hobbies. Get on that.

00:15:50
Speaker 2: Just I also think that people I've just did an episode about this, Actually, I think people also assume some too big to fail thing will happen, even though that's just too big to vote. And the Great Financial Crisis was so much bigger and so much worse, and also very different.

00:16:06
Speaker 1: Like no, the financial scale was very different, which is funny because you know, early on in this one of the things that I found most striking was how quickly, so initially, people were arguing to me that Paul, don't worry your pretty little ahead about this, because these are big boys and girls who are spending all this money. The hyperscalers, the Microsoft's Amazon's Google and everything else. What they spend on their money on should be no concern to you, because these are big and profitable companies. Well leaving a side open air on anthropic, but that are big and profitable companies. How are you to tell them what to do? And then I was so I said, you know, fine, it's so much much. It's my job, but fine. But then it ripped rapidly changed, right because halfway through last year, maybe in the second quarter of last year, suddenly the cash needs of building out these data centers exceeded the cash flow I should say, the unencumbered cash flow of these large and profitable companies, because they have other uses for cash litle. So you started moving more and more towards external sources of capital, private credit, most notoriously, but also a host of other special purpose vehicles and other off balance sheet financing structures, to the point that by the end of the year of twenty twenty five, a data center related debt was the largest chunk of investment grade debt issued in the US, and twenty twenty five tech used to be the no debt sector. They went from no debt to the largest issuer of investment grade and non investment grade debt in the United States last year, and of course all the way along people normalize it and say, you know, it's fine, they're very profitable. And then when it turned out the profits weren't enough to pay for all the data centers, that's also fine, right, So we buill did these things end up being fine? And then of course it all came home to roost to a degree at the end of the year as private credit started sort of got attack from through the side door because of this problems with these very large positions they held in software as the service companies, which it turned out were at least seen is threatened by AI. So that's the first shoe to drop on this stuff. But there's still another shoe to drop, which is the overexposure of private credit to data center related debt, because if you think about it, it's not the consequentiality of it. It is something like the global financial crisis. The more entertaining part of this is they're treating data centers as real estate. They look at data centers as being like apartment buildings with who the hell knows what's going on inside, but they're good for the rent, right, So this is the way they look at at this way look at data centers. But the problem is the thing that's generating the income is inherently deflationary, hyperdflationary, falling seventy to eighty percent year over year. These are tokens. So the idea that you can you're having to pay a fixed obligation these notes that have been issued with respect to the debt to finance data centers with a thing that's following seventy to eighty percent year where year in price. Just try and run an auto company that way with a significant depation.

00:18:47
Speaker 2: But the price of tokens coming down wouldn't affect GPU compute in that way though. Also that price coming down isn't necessarily a result of cost savings, but it's a result of the company's cutting the prices. Isn't the problem that they're also full of these depreciating GPUs as well?

00:19:04
Speaker 1: Yeah, yeah, So there's a double whammy. There's both sides of it, right, So to a degree, if you're working with a model directly through the API, which the largest issuers are, or at least if you're in a production position, you're actually paying an API price ten which is metered at the token level, so you do see those token prices directly. And then secondarily you have the problem that the GPUs themselves, insofar as they are capital investment around which the investments predicated, the depreciation of those items is relatively rapid to it. And the analogy I always make to people is that it really depends on what they were used for historically. So if they're just being used for inference to a degree, they have a longer lifespan. But if they were ever used for training, which is like flat out pedal to the floor twenty four to seven, huge workload, It's kind of like, you know, you had a car that was only driven to church on Sundays and a car that was raised at Lamon's one weekend. They have the same number of miles. I know which car I want. The same thing applies to GPUs.

00:20:00
Speaker 2: Yeah. And the other thing as well is I've really been looking at this. So you've you brought up the wood Mac study, which was awesome. I don't know if you saw the Siteline climate one where it was like, of the sixteen gigabs that were meant to come online this year, only five are actually under construction. I think that there's a big problem with just the speed of the rollout and the upgrade cycle, because we are still going to be installing Blackwell GPUs into twenty twenty seven, if not twenty twenty eight. That's insane that like that it I worked it out as it's like it takes six months to install a single quarters worth of GPUs. But it's like, how at some point Nvidio has to slow not even because of me wanting it to or not, but because.

00:20:45
Speaker 3: Where are they going? Like where are we putting these things? I mean Taiwan warehouses, I guess, yeah.

00:20:52
Speaker 1: Tiewe warehouses.

00:20:53
Speaker 3: Yeah.

00:20:53
Speaker 1: And that's a big problem, is the build out. So this this problem of and of the Northeastern I forgotten WHI Utility in the Northeast just recently put out some data on this showing that something like them data U suggests, which is like twenty five percent of the total commit was actually ever produced and likely will ever be built out. And that's in part because a lot of these things are speculative projects naming no names. There's a very large Texas company that's doing this directly the career that is a very speculative position. We're gonna we're going to power it all behind the meter with nuclear reactors and all these kinds of things. But this is a this is this is a game we've seen back to I mean, the analogy I make all the time is back to Chinatown. This is like Chinatown right where I'm I'm buying up real estate with numbered companies in hopes of securing water rights.

00:21:35
Speaker 3: I saw this.

00:21:36
Speaker 1: I saw this when Jack Nicholson was wandering around Ja Eddies was wandering around in the deserts of California, Chinatown example. So what's happening is increasingly a lot of what's going on under the hood here that's creating the impression of a build out that doesn't exist. Are these things called powered land companies? So powered land companies are these speculators they don't call themselves that, who look for strategic location. We're using numbered companies. They can purchase real estate that has access to peering points, so high speed interconnection.

00:22:06
Speaker 3: To the what is a number of company as well? I'm really sorry.

00:22:09
Speaker 1: So a company that doesn't make it obvious to who the actual direct owners are, so it's more clear at what their purpose is. So I you know, think of it as like Cayman Islands.

00:22:16
Speaker 3: But it doesn't.

00:22:17
Speaker 1: So the idea that you're trying, you're trying to at least loosely obfuscate who the what the ownership and purpose are. But even that's less important. The idea though, that they're buying on a speculative what basis This tracts of land could be you know, hundreds of acres in some location that has access to power, access to water, and potentially access to a peering point, a high speed interconnection point to the broader Internet. And then they then they lock that up. And then they go out and say, okay, I've got this position. You guys need to build out more data center capacity, talking to the hyperscalers.

00:22:47
Speaker 3: Look at me, you got.

00:22:48
Speaker 1: Nowhere else to go. I've locked all this up early on, kind of like locking up the water for the orange grows?

00:22:53
Speaker 3: Right? Is that is that widespread? Is that widespread?

00:22:56
Speaker 2: Think that's so bad because I mean, this whole time I've been looking for the speculative pop I will I fully admit that's been it. If it's the land, that's not great for anyone involved. But even then though gpu is still being sold like that, that's the real.

00:23:12
Speaker 1: So there's a game, right. But part of the problem is, and there's was a great piece from trend Force, I think it was the other day, one of the market research firms. In this they're pointing out how how how many firms are now buying LTA's long term purchase agreements because they they're being told that if they don't lock in demand now, lock in now, they won't get product in two years. So this is the other layer that a lot of what you're seeing is purchasing isn't completely speculative by people who are worried that they don't lock in a long term purchase agreement now, they will never get supplied later, so.

00:23:42
Speaker 3: That their approaches.

00:23:44
Speaker 1: I'll worry, I'll worry my about that other stuff later on, but for now I need to sign up. So that's the other piece that a lot of people miss here is a lot of the demand now is increasingly tied into these sorts of long term purchase agreements which have nothing to do with actual units being shipped.

00:23:56
Speaker 3: The mob boss thing again, Yeah yeah, yeah.

00:23:59
Speaker 1: But I was still getting out. I told someone this the other day. Who was I talking to you? I have the Wall three journal I was talking that was saying like it's a it's kind of like the old mafia threat, Like really nice AI market you have there if something happened.

00:24:11
Speaker 2: To it, Oh you want you don't, I guess you don't want Vera Rubin anymore.

00:24:17
Speaker 3: I guess we'll have to give.

00:24:19
Speaker 1: It's exactly like that. So this is the thing. So those two pieces are really important because it creates the impression of unit growth where unit growth doesn't actually exist because it's predicated on locking supply in the hopes of something later. But at the other side of it, you've also got the speculative lamb component. There was a company the other day, there was a great Bloomberg story about it who raised I think it was three billion dollars in junk bonds for exactly this purpose. So this is highly speculative stuff that's actually maybe no, no, not terrible. So just literally last week, uh.

00:24:51
Speaker 3: It was called tracked.

00:24:53
Speaker 1: I think it was called t R A C T.

00:24:54
Speaker 3: Yes, tracked. I remember this one. This is the this is the insane thing.

00:24:59
Speaker 2: They're still able to set like raise those bones though, like they're still able to get the money.

00:25:04
Speaker 1: Well that's because again this is this, this is the insidious problem. Here is there's this idea that if I'm successful, my counterparty, the counterparty in the data centers, they're good for it because Microsoft Google. So instead of having a bunch of dodgy you know, Florida strippers or something on the other side of this, like in the financial crisis, what I've got on the other side, my counterparty has a high credit, very focused, very small group. It's the Microsofts and Googles and others. So people are willing to take much crazier risks because the counter party looks so good from a credit standpoint, which is very different from what happened in the financial crisis, where I wouldn't have issued junk to create something that was going to be purchased by you know who knows. But if it's Microsoft, Google and the other hyper skills the other side, I'm like, you know, what, if this works, they're good for it. But what if it doesn't work, well, then of course you went up with a lot of room, a lot of extra buildings for you know, laser Tag or something like that.

00:25:56
Speaker 2: I'd I'm excited about the Laser Tag arena arena in the future, we have just just America's the Laser Tag Capital of the world.

00:26:04
Speaker 1: That's right, We've got a lot of extra space for use storets and laser tag Because again, the only thing we know for sure is that the the efficiency of inference. If you buy the argument that we're transitioning rapidly to inference, the efficiency the inference is going on is rising rapidly because of things like distillation. Models are shrinking, chips are becoming more efficient, there's less less memory.

00:26:39
Speaker 2: Required, because that's something I've The people that have told me inference is getting more efficient are usually referring to Nvidia demos based entirely.

00:26:48
Speaker 3: On d Yeah, that's a ridiculous question.

00:26:51
Speaker 1: Look at companies like Fractile out of the UK, an interesting company with I think some really groundbreaking inference technology that will be ship this year. Look at a company like Talus in Torontol. Again, so Talus is a good example. They're doing some innovative stuff showing So a high speed inference chip today might do a few hundred tokens per second, that's so considered a relative leaving aside the wattage required, that's a relatively high speed token UH producing chip. So Talus is demoing sixteen thousand tokens per second so we're seeing step function increases at lower power from some of these next generations silicon vendors. Granted, they're not going to dominate the marketplace, but the idea that we're not going to see step function changes that we can project into the future based on what we've seen in the recent past is just dead rock.

00:27:40
Speaker 2: But the thing is what models are they able to do that with? Because again, a lot of I mean, all of the benchmarks we see are based on open source models, because open source models are open sourcest ones they can test on.

00:27:53
Speaker 3: I feel like everything every time I get any kind.

00:27:55
Speaker 2: Of leak out of azore or AWS about these models, it's I have a Microsoft personally topic.

00:28:03
Speaker 3: It's like four to.

00:28:04
Speaker 2: Six, sorry, four to twelve GPUs, one generation of the smaller reasoning model. Oh four I think it was, Yeah, And that's just for one generation over several minutes that someone's doing a particularly different coding task. It doesn't feel like these inference chain things will trickle down there, So maybe it will be the future of llm's. Is this small industry run on these smaller chips or something like this.

00:28:30
Speaker 1: I think you're gonna see all of the above thing. Well, let's say, for example, we're seeing inference happening inside of evs where it's we're doing. I'm doing rapid ingestion of video tokens for the purpose of deciding whether I'm going to back into my neighbor's trash can and these kinds of things. Language models, Oh, absolutely, because you can you can imagine I'm ingesting all of that the video. I'm gonna have a lot more granularity with respected the tokens flowing back to me, and I can do more with it and know more about what's happening in my environment. I think most of video ingestion is gonna move towards tokens, but done on small language models, very very small stuff.

00:29:04
Speaker 2: We've already had edge compute AI vision stuff for like a decade or more.

00:29:10
Speaker 3: Like I was working on stuff like that in the twenty fourteen.

00:29:12
Speaker 2: It just feels like a lot of this is trying to make tokens through stuff that we already kind of do.

00:29:20
Speaker 1: It is but the thing I and again you know, I'm deeply in this skeptical camp here. The thing I will say in favor of tokens in terms of absorbing a lot of this stuff is that it makes it all less ad hoc because you now have this sort of universal commodity for ingestion and production of information. That's that makes things a little more interesting because I can now abstract away some of the hard problems of video processing. I can abstract away some of the hard problems with speech synthesis because they all kind of disappear and become in a sense of this universal token. And you know, to a degree, I think that's true, and I think it will lower the barriers to more people playing in the worlds of you know, video and speech and other places. But that doesn't create the kind of marketplace that people who are pushing huge numbers of tokens run on frontier models.

00:30:06
Speaker 3: Why this is just edge stuff, cheap and dirty.

00:30:08
Speaker 2: So the problem with tokens as a commodity as well is it's very hard to know, like one toe like a million tokens per million tokens. It's impossible to actually measure how longer tasks, how many tokens a task will use because of the inherent I'm reliable of large language models, and it's like so it's hard to weave it. Like right now you're seeing with this, have you seen this people complaining about anthropics rate limits.

00:30:34
Speaker 3: For example, I'm not sure all the time I see it.

00:30:37
Speaker 2: Well, right now people are mad that they can only spend a thousand dollars on a two hundred dollars a month plan. But the thing with that is people very clearly do not know how much to how far a token goes or a million tokens goes, like they it's it's difficult to evaluate and measure that, which feels like kind of economic poison at some point because if you can't say how much a task will cost, you don't have a miles per gallon, like sixteen thousand tokens per second. Well, you could do inference fast, but if a customer can't afford it, if a customer can't actually reliably say I'll be able to use this in this way, what uses tokens is a measure, but I mean I know what they used for a measurement. But it's like you can't say what even a million tokens might do.

00:31:19
Speaker 1: No, you can't. But that's kind of innate to the world of like a true a true commodity. I don't know how much you're going to put in your car. I don't know it. Really that becomes an engineering decision for you, not a production.

00:31:30
Speaker 3: Decision for me. Right, So yeah, that makes that you have to sumporate.

00:31:34
Speaker 1: Those two pieces and so that I don't can't tell you, here's so many tokens it will take for you to do X. That's no more my job than it is for me to tell us that, you know, a copper mind how many how much copper it is going to take.

00:31:44
Speaker 3: For g I'm in a particular car, so know how much? Yeah, but they know how much. I get what you mean. It's like they know how much copper they need to use.

00:31:52
Speaker 1: But then and then they'll make constrained, you know, constraining decisions whether they'll say, I only want to use this much copper because copper is really freaking expensive, and so you know, we're gonna like I just saw I think it was really in the other day, said they'd cut out like I don't know, like ten miles of electric cable inside their cars, which seemed ridiculous to me, but it was again because of the price of copper. So there's yeah, so they turn into and these things turn into engineering decisions. Right now, we're in this kind of subsidized wild wild West where everyone thinks it's a land grab and they're so they're subsidizing it to a degree, and people are over using tokens. Oh, I've had a model running for three days and it's doing all these agentic things and it's like, well, what are you trying to create out of it? And it's like, I don't know, it's some nonsensical things. So it's people are being subsidized to do non economic behaviors to an incredible degree right now. And I find that remarkable, which is a statement about this kind of land grab mentality among the frontier model vendors, and and Dariol has been real. Darioeminad in the Tropic has been very upfront about this. Is he believes that we are in a land grab mode. There's only going to be a couple of frontier models vendors left standing, and so we need to make sure that you know, we're the dominant provider, if you will, of you know, coding harnesses in frontier models. Now, I think that's something Snowmer too, but that's another.

00:33:00
Speaker 3: Yeah, I think.

00:33:01
Speaker 2: But my core economic I mean one of the many core economic things, such as it's totally unprofitable, My thing right now is these rate limit changes are more severe on an economic level than people give credit for just because of the economics, but because.

00:33:15
Speaker 3: Of the habits.

00:33:16
Speaker 2: If you believe, like the way I analogize it is like if your car can drive fifteen miles and then one day it can only drive three, can you get to work?

00:33:25
Speaker 3: Yeah? And is it because if you.

00:33:27
Speaker 1: Bought a house predicated on being able to be drive the fifteen miles to work, right, So it has it has externalities. This is outside.

00:33:33
Speaker 2: Consequence, yes, but the thing is you're paying. They've trained everyone in this land grab to act in a way that doesn't make sense long term. And I'm not I don't think they can become profitable. I actually truly don't think that there's an economic way for it to happen. But they've trained people to use the product in a way that doesn't make sense. Like it's not even a oh they can charge more. Your habits are not built for this.

00:33:56
Speaker 1: That's exactly right. So I have a wild eyed theory. Are you ready?

00:34:00
Speaker 3: Don't go go?

00:34:02
Speaker 1: So the IBI theory as the first of the first frontier model company to abandon Frontier models wins.

00:34:08
Speaker 2: What do you mean?

00:34:10
Speaker 1: So my theory is that all of this stuff, most people in a Pepsi Cooke challenge kind of way, can't tell the difference. They claim they can, but they can't actually tell the difference between most frontier models for a typical test. Certainly normals can't. Coders claim they can, but if you actually do it in a blind way, most of them can't tell. This is just ego, and so increasingly what people see instead is these coding harnesses, these tools like quad code and codex and open code and all these sorts of things. So most of the value they see, and most of what they actually think of as the model is just the harness, and that's where most of the innovation is happening. So my argument is, and that's why it was so dangerous Foraranthropic this week when they accidentally did a whoops and released quad code. Most of the values in the harness. And so the first company to say, you know what, we don't need to spend this kind of money anymore on training new models, because we're going to just sit on top of models from all of these loons who are out there spending crazily on new frontier models that aren't improving very much anymore now, certainly not like they were four or five years ago, and that they will be rewarded for that, no different than saying, you know, I've let go all of my employees, or I've decided to stop spending money on hydroelectric dams. You've cut cap X, you've cut You've made your business more financially appealing by taking away the single biggest piece of cost because you're recognizing that the world has changed, that I'm not getting incrementally as much value for dollar on training a model as I was five years ago. And that's very clear in the data if you look at any of the composite benchmark models getting away from like can they solve math problems? But literally real world composite models, it's been a sharp decline from like, you know, eighteen percent year over year improvements in models to like four or five percent at vastly higher costs and more dollars and more time. So that really matters, and so my completely will never happen. Theory is that the winner here is the first one to stop doing it.

00:35:49
Speaker 3: Isn't that just describing cursor?

00:35:53
Speaker 1: So cursor is an interesting examp. So cursor doesn't actually embrace the full idea of being a harness across all of these white collar applications. They're still kind of trapped in a coding world, and so the idea So if you're right, so I just think people get trapped in coding because it was the first place this stuff emerged. So you have to think about, like Cowork is a good example of at least an attempt to break again, escape containment and get out of the world of coding and say, okay, this is actually for all white collar workers. Like I watched people struggle with Claude pardon me.

00:36:22
Speaker 3: But it didn't work.

00:36:23
Speaker 1: It doesn't work, but it's the least it's directionally the right idea if you buy my theory that all of this stuff is commoditizing so fast and there's a loser's game financially, that maybe the right sort of game theoretic strategy is to be the first frontier model company to stop making frontier models. And in a weird way, Apple kind of showed the way right because early on they were getting pillaried for why is an Apple spending more on AI? Why does Apple not have a model? And now of course it's reverse where it's like look at Apple, They're so smart.

00:36:50
Speaker 3: They're gonna come smart lail without that TAPEX. I don't know.

00:36:53
Speaker 2: I just feel like I feel like what you wore describing as just AI modal wrapper companies. Yes, but I so My whole thing is unless someone is able to break out of coding, there isn't really a hope for any of this because to this point, every time I read about an integration with like a Goldman sax or somewhere, yeah, I can never actually find out what it does.

00:37:15
Speaker 3: I can never the further you get into the region, I have a good one for you.

00:37:19
Speaker 1: So I was, I was talking to a very large investment bank the other day about how about they're you know, prodigious AI integration efforts, and so they built it out. They they told me across equity research, sales trading, and investment banking, and they asked me which one do you think has seen the greatest benefit? And I said, oh god, I said, I used to work on the cell side. I said, my first instinct is none, that they're all lying to you. But my second instinct is, I'll say equity research because you know, no one likes to build spreadsheets, and maybe it helps them build spreadsheets, and they said no. So the answer, of course was investment banking. And I said, why investment banking? These are like knuckle dragging dinosaur. Okaye, man, what are they doing? And he said, the answer, of course is the main thing junior investment bankers do is build. Well, they get shouted at, that's the main thing they do. But the second thing they do after being shouted at is they build pitch decks for companies that don't want them. So they build a pitch deck because a partner wants a pitch deck built for some random company somewhere. And that's a pain in the ass. And so now that used to cause all these sleepless nights and blah blah blah, they're doing them all with AI. So junior investment bankers love AI because it lets them do this completely unproductive, largely and consequential task of building pitch decks for companies that don't want the pitch deck. And so that's These are the kinds of applications that have really minimal economic value and yet sort of superficially appear really exciting because if I'm someone who otherwise had to stay up all weekend building a PowerPoint deck to pitch to some random small cap company and I'm like, yeah, this is terrific. So that was the answer. It was really interesting, Yes.

00:38:47
Speaker 2: But that's also kind of worrying because like, the best example we have for this thing that has taken over everything at least optically, even though asn't in economic terms, is like, we can do powerpoints kind of for we.

00:39:02
Speaker 1: Can do pop power points for junk bond raisers for microcap companies way better than we used to.

00:39:08
Speaker 3: Wow. And it's like helping junior analysts.

00:39:10
Speaker 2: So it's like, are you really the time they're saving is just lowering their workdays from fifteen hours to eleven.

00:39:17
Speaker 1: Well, and they're still being shouted at unfortunately for them, but you know, that's the way it goes.

00:39:20
Speaker 3: That's that at the bottom of the job.

00:39:22
Speaker 2: Part of the Joel Paul, It's been such a pleasure having you. Where can people find you.

00:39:27
Speaker 1: Pokadrowski dot com.

00:39:29
Speaker 2: Thank you for joining us, and yes we'll be back with the monologue this week. I'm of course ed Zeitron. Thank you everyone for listening. Thank you for listening to Better Offline, The editor and composer of the Better Offline theme song is Matasowski. You can check out more of his music and audio projects at Matasowski dot com, M.

00:39:53
Speaker 3: A T T O S O W s ki dot com.

00:39:58
Speaker 2: You can email me an easy Better offline dot com or visit Better Offline dot com to find more podcast links and of course, my newsletter. I also really recommend you go to chat dot Where's youread dot at to visit the discord, and go to our slash.

00:40:11
Speaker 3: Better Offline to check out I'll Reddit. Thank you so much for listening. Better Offline is a production of cool Zone Media.

00:40:18
Speaker 2: For more from cool Zone Media, visit our website cool Zonemedia dot com, or check us out on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.