SAM JARRELL Michael, good morning, and I have a question for you today. How much do you think the last AI prompt you wrote cost? MICHAEL BIRD I've absolutely no idea. I mean, from a user perspective, it sort of basically looks free, doesn't it? So I'm gonna say, practically free. SAM JARRELL Practically free. That's, I think, a, a fair assessment 'cause, for most users, they don't really get to see the cost unless you're paying for a subscription. But practically free could suddenly become problematically expensive if it's multiplied up across, an entire organization, right? MICHAEL BIRD yeah, it does all add up. and I suppose from an organizational perspective, planning the finances of your AI strategy is pretty important. So this week we are starting a two-parter exploring the subject of tokenomics I’m Michael Bird SAM JARRELL I'm Sam Jarrell MICHAEL BIRD And welcome to Technology Now from HPE. MICHAEL BIRD I think it's fair to say that AI is our favorite topic on the podcast at the moment SAM JARRELL it's up there with cybersecurity and self-driving, I think. MICHAEL BIRD Yeah. Well, today we're gonna be looking at how you measure AI usage because while it might feel like it, it is definitely not an unlimited resource SAM JARRELL Exactly. And a lot of things like chatbots can have different tiers that give you different amounts of usage. So there has to be a way of keeping track of how much you're getting though MICHAEL BIRD Yeah. And, and the way we measure AI is using things called tokens, but not all tokens are created equal. So to find out more, I met with Dan Waibel, Global Chief Data and AI Officer at HPE. and because tokenomics is such an important topic that we really wanted to cover, we're gonna do this, across two episodes. So in this episode, Tokenomics Part One, Dan began by telling me what exactly a token is. DAN WAIBEL So the easiest way to think about it, I think, is a token is the smallest piece of something a model reads or writes. So for text, it's usually a chunk of a word. Common words like the or data are one token. Longer or rarer words get chopped into pieces. Numbers get chopped up. Code gets chopped up a lot. so it's not exactly a word. the rule of thumb I use is about 1,000 tokens for every 750 words of English. So a typical 400-word answer from a chatbot is a bit over 500 tokens. And here's the bit that matters, if you run a business. Tokens are how AI is metered. Every question you ask goes in as tokens. Every answer comes back as tokens, and someone's counting both. So it's really the meter on the wall MICHAEL BIRD so how does the model know to chop a word into two tokens? DAN WAIBEL It's really how the model processes, units. So an easy way to think about it is every three to four to five letters will be broken into a token, and there's input tokens and there are output tokens. MICHAEL BIRD so a token is sort of the fundamental unit of an AI model. It's like the atoms to an AI model DAN WAIBEL That's correct. Yep MICHAEL BIRD And so when companies talk about creating tokens, what does that mean? DAN WAIBEL So again, tokens are really just a way that we measure input and output into the model. we need a way to measure that, for a number of reasons. And I think really one of the main drivers is cost, of course. So this whole concept of tokenomics, and that's really how models operate, frankly MICHAEL BIRD Okay, so tokens relate quite significantly to the cost of running an AI model or an AI workload DAN WAIBEL =, there's input costs, there's output costs, There's aggregate cost every time you access an LLM. And the, the output cost is about six to eight times as much as the input cost. And the reason why is when the model ingests text, it breaks it down into tokens, but it ingests it all at once. When it produces output, it's producing it token by token, and that drives a higher level processing, and that higher level processing again translates into a higher token cost. MICHAEL BIRD So let me just check that I understand what you're saying here. when you're inputting a block of text, or whatever, it's just understanding what you're looking at. But because it's then processing that information, it's doing something with that information, because there's the processing involved, the cost is higher as from, from an output perspective? DAN WAIBEL That's correct. MICHAEL BIRD okay. So you said this word tokenomics. what do we actually mean by that? DAN WAIBEL it's the economics of AI consumption. What you pay per token, how many you use, what value you get back. it's really the bill, the price times quantity. and everyone's been watching price, and almost no one's been really managing quantity. And to me, the quantity has two surprises. First, different kinds of AI work use wildly different amounts of tokens. And second, who is using it matters more than you'd think MICHAEL BIRD Yeah, because in particular large language models hit the, public consciousness and it was offered to the public as free to begin with. but actually, as organizations, we're now having to start to think about the true cost of running an AI model, and that true cost, is on a, per token basis. DAN WAIBEL That's been a big shift. the hook was pro bono, as you mentioned. and now I think, many of the enterprise tools and platforms that are available have moved to consumption-based billing And, much of that is driven by the increasing costs of AI, but I think ultimately this was always the plan in the long run. MICHAEL BIRD And consumption billing is really an, an interesting one, isn't it? Because, a token feels like... it's a very small, unit of measure, but you don't necessarily know how many tokens you're using from an input or an output perspective, do you? DAN WAIBEL I agree. I think, it is a bit opaque. one of the things that, that we focus on at HPE is ensuring that everyone understands token usage. So, we're driving transparency through our tools and platforms. and as you use tokens, we're very clear and transparent about how many tokens, you're using, because we, do have quotas and I think the other piece of this that's really important as we move forward is agents and agentic, because it's already driving, significant token costs. I would say, running an agentic workload could be anywhere from 10 to 100 times more expensive than just a typical chatbot. So you can imagine as we stand up more of these agents and these multi-agentic systems, the token costs are gonna skyrocket MICHAEL BIRD Yeah. And, do you think we'll see, a change in the way that users are perceiving their usage? Like, when you use a a large language model, do you think the user will see, "You have 20,000 tokens for use today," and it'll sort of tick down as you're using it? DAN WAIBEL I think that will become the norm at some point, and we're already moving in that direction. You know, if you're leveraging, our core internal platform, Chat HPE, we have a real-time counter of tokens input and output for you as a user. So you know where you stand, um, quote-unquote, you, you know how much money you have in the wallet, right? MICHAEL BIRD Yeah. and I remember there were stories, maybe nine or 12 months ago of large organizations wanting to really encourage the use of AI within businesses. And, they were targeting their users on essentially who can use the most tokens. Are we starting to see a shift in the way organizations view token consumption as a measurement? DAN WAIBEL Absolutely. So if you go back to the beginning of the calendar year, there was this trend around, what's called token maxing. So particularly amongst the tech companies, where companies started tracking who's using the most tokens and almost as a reward. I think, we've really shifted away from token maxing, and I think really what we need to shift towards, really as an industry, is cost per value. I think that's the realization that's come about. And yes, there's a number of stories circulating around how actually some, some large tech companies have burned through entire budgets for the whole year in one or two months, which is incredible. but since I think there's been a major shift back to value versus just who's using the most tokens. it's almost like a utility. it should be thought of that way, like power, electricity, cloud. you wouldn't run a data center without knowing which workloads draw the electricity, right? So same here, I think with tokens MICHAEL BIRD And, when we're thinking about tokens, like what defines the cost on a per token basis? DAN WAIBEL It depends on the type of workload. So if you're simply chatting with an AI and, it's more of a, a Q&A session, that's gonna be much, much lighter than, for example, if you're kicking off an agent workflow or a multi-agentic workflow, that's gonna be much more costly and much heavier. or for example, if you're generating an image or generating a video, the token cost is significantly higher. MICHAEL BIRD and when we're thinking about output tokens, just so I can wrap my head around it, is a token a fixed amount of compute DAN WAIBEL So the, the tokens are fixed, but the cost per token varies per model And depending on the model that you're using, those costs can range significantly. So if you're using the latest frontier model, the cost might be 10 or 20 times higher than a model that was considered frontier six to nine months ago. MICHAEL BIRD And is there a agreed definition of what a token contains or, how much compute a token is or how much energy a token is? is it something that can be commoditized? DAN WAIBEL I think the definition is very standardized. but ultimately, if we drive back towards value, that's where there's significant debate around how do you choose and leverage the right model for the right task MICHAEL BIRD And has the price of tokens, gone down or gone up over time? DAN WAIBEL So the price of tokens have gone down significantly over time, particularly if you think of from a capability perspective. LLMs, I think every about six months or so, they're doubling in terms of their capabilities. So if you think of it from a capability perspective, the costs are actually going down significantly because the capabilities are doubling every six months. That being said, we're also using many, many more tokens than we were even six months ago. So, this Jevons paradox of as a utility becomes cheaper, it's used more. So I think that's really what we're running into right now. And again, much of that is driven by agents, 'cause agents are chewing up tokens much more quickly, than just pure chat MICHAEL BIRD are these models looking to also become more efficient? you mentioned utilities. If I think about, electricity coming into my house, 50 years ago, I'd be using incandescent bulbs. nowadays I use LED light bulbs, so the cost to light a room, has decreased. is there also that consideration of how can we use less tokens to achieve the same output? DAN WAIBEL Absolutely. And, that's determined not just by the model purely. there's context engineering and harness engineering that goes into really optimizing and becoming more efficient. there's multiple layers to this beyond just the core model itself. And then of course we have open models and closed models MICHAEL BIRD Can you just talk about the difference between those, open and closed source models? DAN WAIBEL Yes. So let's start with closed source models. So these are typically your, quote-unquote, "frontier models," and these are the most powerful models available at any given point in time. and a closed model, a frontier model, you really rent that capability, you rent those tokens, you send your tokens to someone else's service, and you pay by the token. An open-weight model you can download and run wherever you like on your own infrastructure or someone else's. two years ago, that was an easy choice. if the task was hard, you needed the frontier. What's changed is how fast open-weight models have caught up on the complex tasks the frontier models used to own exclusively. So things like reasoning, coding, working with long documents, not on everything, but on a lot. And that matters to the bill in really a few different ways. Cost per task, because a smaller open model doing a job well is a fraction of the price where it runs, because open weight means you choose the location, and of course, lock-in, because you're not tied to one vendor's roadmap or one vendor's price list. So there's a lot to consider, when you look at open and closed models. MICHAEL BIRD so,. the closed source models are, those front-end models. that have essentially every single piece of the internet, every single bit of knowledge they can scoop up into these, massive models. but if you are doing a task that maybe relates to your business or a very specific part of, an industry, you actually don't need the world's knowledge to be able to answer a question in a particular area. So actually, those models can be more efficient because there aren't as many parameters to have to work with. have I got that correct? DAN WAIBEL That's true. small language models can be more efficient than large language models, and domain-tuned or domain-specific models can also be more efficient to run. than a general model MICHAEL BIRD is that different to open and closed source models? So is an open source model not necessarily a small language model? Can it also be a large language model? DAN WAIBEL That's correct. a open source and a closed source model can be small or large MICHAEL BIRD Right. so what is then the difference to an organization? the fact that you can run it on your own hardware has its own benefits. Have I got that right? DAN WAIBEL That's correct. I'll talk a little bit about how we approach this at HPE. a few years ago when we first launched generative AI internally we were tapping the frontier, and we were building our platform internally, leveraging closed source models. But then open source models, started gaining some steam, and we decided to build a hybrid architecture where we can route AI workloads, these tokens dynamically to open weight models or open source models or closed source models. And, that routing, we've added significant intelligence there so that we wanna use, our on-prem capabilities as much as possible because that's gonna drive the highest level of efficiency. But we also wanna be able to bounce out to the cloud, for frontier models and for more complex reasoning tasks. So that's how we've built this intelligent routing. MICHAEL BIRD So that was, part one of our interview. And, it was just so dense with information. There was so much in there that I'd love to talk about. But, the overarching theme is that, over the last few years since, the end of 2022, AI has felt, both the most expensive and the cheapest thing, in the world. you hear these staggering sums of money that's being spent on hardware, but actually from a user perspective, it's really easy just to, jump onto a window and ask an AI model a question. SAM JARRELL you could call it, the AI paradox. it's the most expensive and cheapest thing in the entire world right now. The only time cost has really ever come up for me is I very briefly, paid for a subscription to a model, when I was doing my wedding planning last year. just because it was helpful with all of that kind of stuff to have, unlimited usage of it, but it was still all chat-based. What was interesting to me about this conversation is the tokenomics of it all and agentic AI, and that they're chewing up a lot faster than the average person is. Because I think for a lot of stuff, the average person may still want to do it themselves to some degree, and then augment their thinking with the power of AI. Do you think it would be more helpful to have a counter, like Dan was talking about, showing you how much usage you had for the day, or if there was a counter telling you how much your agent had to use and if you had to share MICHAEL BIRD I think that's the crux of it. we've lived in this world of there are no limits. AI is this sort of endless resource that will solve many of the world's problems. And, I think Dan used the word token maxxing, which, I think we alluded to in the interview. there were these stories of these companies that were targeting their employees, to use as many tokens as possible, and they had a leaderboard, and it was like whoever used the most tokens was, employee of the month or whatever. but actually that doesn't take into account the cost of that. SAM JARRELL Value maxxing. MICHAEL BIRD And I suppose like exactly like value maxxing. You heard it here first. I wonder actually if organizations will think, "Okay, well, actually how much does this particular task cost versus, getting a human to do it versus getting an AI model to do it?" it sort of feels like we've understood that this is an incredible piece of technology, but maybe we're starting to think about actually how much does it cost to do it. And, Dan used the phrase, to create this particular bit of value for your organization or to complete a task or whatever. what is the actual cost of doing it? Because these tokens cost money. They're not free. And, where does it come from? I don't know if that answers your question, Sam, SAM JARRELL Well, it goes to a deeper question for me, which is like if we're talking about the organizational level, who should be responsible for managing consumption? is it the individual, the technology team? Is it like the, the finance team? it's interesting to think about like where the line is going to be eventually for determining like how you dole out tokens. MICHAEL BIRD the first few years it'll be a bit of trying to figure out who needs the most tokens, but over time it'll be, like developers need more tokens because they're using them to create this sort of value and, different departments use different things. but I also think there will be a conversation of, do you need to run that query or that question does that need to be run on this large language model which costs more per token versus this smaller or, run on our own hardware model where the per token cost is cheaper. And I think Dan alluded to this, but having some sort of, orchestrator to be like, "Okay, this can just be run locally. In fact, It doesn't even have to go through an AI model. It can just be run through a dictionary or through a translator. SAM JARRELL Yeah. to be honest, in some ways I think it's good for people to have to do a little bit of searching and think critically about where they're getting their answers and things from, um, versus what has become natural is replacing search with just asking AI search a question and then being like, "Oh, okay, so that must be the answer." MICHAEL BIRD Yeah, yeah. MICHAEL BIRD now Sam, we talked about, open and closed models earlier, and we didn't really have time for this in the main interview. But one thing which I found particularly interesting is that they often come from different countries. and in the case of open versus closed models, they could be considered to be part of the AI arms race. So I'll let Dan explain this a bit more DAN WAIBEL most of the open source or open weight models are built by Chinese, or China-based AI labs. And they're looking to compete with the closed source models or the closed source labs, which are all US-based, and that's their way to compete because they don't have the capital and they don't quite have the depth of talent, although they're catching up very quickly. it's an interesting question as to why more US-based, open source or open weight models are not available at this point. But you'll see different players, depending on how they benefit, may stand on different sides of that equation, right? the difference in the two, that's coming together very quickly, right? particularly for coding, which is of course really the core heaviest weight agentic and multi-agentic AI workload out there right now SAM JARRELL Okay that brings us to the end of Technology Now for this week. Thank you to our guest, Dan Waibel And of course, to our listeners. Thank you so much for joining us. MICHAEL BIRD If you’ve enjoyed this episode, please do let us know – rate and review us wherever you listen to episodes and if you want to get in contact with us, send us an email to technology now AT hpe.com Subject line, Sam? SAM JARRELL Value maxxing MICHAEL BIRD Ooh, value maxxing. I was gonna say Jevons paradox. you can put either in the subject line, that's fine. and don’t forget to subscribe so you can listen first every week. Technology Now is hosted by Sam Jarrell and myself, Michael Bird This episode was produced by Harry Lampert and Eva Higginbotham with production support from Nik Damarell, Beckie Bird, Alissa Mitry, and Jenessa Ayache. Our theme music was composed by Greg Hooper. SAM JARRELL Our social editorial team is Rebecca Wissinger, Judy-Anne Goldman and Jacqueline Green and our social media designers are Alejandra Garcia, and Ambar Maldonado. MICHAEL BIRD Technology Now is a Fresh Air Production for Hewlett Packard Enterprise. (and) we’ll see you next week. Cheers! SAM JARRELL Bye y’all