00:00:01:00 - 00:00:18:01
AJ Kuftic
Hello, everyone, and welcome to today's webinar where we're going to be talking about deep seek and the effects that it's having around AI. And honestly, reasoning models in general. If you have any questions, please go ahead and scan this QR code. It's going to pop up here in the top right corner. That'll give you the ability.
00:00:18:01 - 00:00:36:17
AJ Kuftic
Top left corner, actually, that, allows you to submit questions. Now, we did get a, wonderful bunch of questions as part of the registration. So thank you for putting those in. And we'll get to those, after I speak with, two of our folks here at Expedia who have a lot of, experience and understanding around deep sea in the AI models.
00:00:36:19 - 00:00:53:20
AJ Kuftic
And, I got a chance to sit down with our manager of innovation and platform development, Mike Caruso, to talk through what Deep Seek is doing. So let's go to that now is Mike Caruso, who's one of our principal technologists looking into AI. And Mike, how are you doing today?
00:00:53:22 - 00:00:55:02
Mike Garuccio
Pretty good. How are you.
00:00:55:04 - 00:01:09:23
AJ Kuftic
Doing? Great. We've heard a lot, over the last few months about this thing called deep seek. There's a ton of buzz around it, but how? Like, what is it like? What is deep seed? Why is everybody talking about this all of a sudden?
00:01:10:01 - 00:01:29:13
Mike Garuccio
So Deep Seek is a Chinese company that recently released, two new large language models. So what they call deep V3 and deep seek R1, V3 is a traditional language model, kind of comparable to GPT like for O. And then R1 is they're what they're called reasoning model, which we'll cover what that is in a little bit.
00:01:29:15 - 00:01:49:19
Mike Garuccio
And that's kind of comparable to the GPT. There OpenAI is GPT oh one. And the real reason that they got, so much attention was basically three different factors that made them unique and new, or at least appear that way. The first one was they basically landed on the top of all the common AI benchmarks out there.
00:01:49:19 - 00:02:09:07
Mike Garuccio
So at least on paper, they had the best models in the world. And they were open late. So other people other than them could run them. They also had a very low claim training cost, which sort of threw the stock market into a little bit of turmoil for a few days because, it called into question a lot of the investment being made in GPUs.
00:02:09:09 - 00:02:30:03
Mike Garuccio
A lot of really both of those things have been pulled back. There's a lot of questions about whether that, benchmark result is due to overtraining or overfitting to the benchmarks, as well as whether those training costs are real or if there's things we're not being told, going on there. So a lot of that initial worries been pulled back.
00:02:30:05 - 00:02:44:00
Mike Garuccio
But what you're left with is really a model that has, very, very low inference costs. If you want deep seacoast it for you, and then lower than a few competitors, inference costs, if you do have someone else other than deep seacoast it for you.
00:02:44:02 - 00:03:07:23
AJ Kuftic
So I think a lot of the buzz that we also get around deep seek is around the fact that it's coming from a Chinese company, and there's always an inherent concern when China is brought into the equation, because most American companies don't want to send data to China. So how are organizations able to take advantage of what Deep Secret is doing to kind of avoid those concerns around the data security?
00:03:07:23 - 00:03:12:18
AJ Kuftic
You kind of mentioned it a little bit around being able to run that model elsewhere. But how does that work?
00:03:12:20 - 00:03:44:13
Mike Garuccio
Yeah, so really there's two different components here. So there's the model, the deep seek release that anyone can go download. And if you have enough GPUs available to you, you can run and then there's deep seek as a service that deep deep the company is hosting. So that service is really the thing that would give you problems in terms of compliance and data security, because regardless of where they're actually hosting it, that model or that service is run by a Chinese company, and you have all of the inherent potential data security problems that are wrapped around that.
00:03:44:15 - 00:04:03:14
Mike Garuccio
Especially since the last I checked, their terms of use does not include a we won't train on your data. We won't use your data clause. Like most of the providers in the West do or will offer you. But there are a number of providers out there that have gone and had downloaded that model and will host it for you and give you access to it.
00:04:03:16 - 00:04:21:02
Mike Garuccio
The model itself, there aren't any real concerns with as far as data security goes. There's not going to be some thing happening inside of there that's going to cause your data to leak. And so that is a good option if you really do want to run deep. Zeek, it can be tricky to find a provider that will meet your compliance needs.
00:04:21:04 - 00:04:43:07
Mike Garuccio
But that's obviously something that they can help out with and try and make sure, make that available to you without that headache. The other option is since the release of DPC and this is, you know, a few months on now, most of the other providers have released comparable models that will give you a similar level of performance at a similar price point without any of those concerns, because they're from the well-known vendors in the space.
00:04:43:07 - 00:05:03:08
Mike Garuccio
So that's, Google with their, Gemini 2.0 thinking, cloud with 3.7, thinking. And then, OpenAI with both O three mini and then GPT 4.5, although that one is much more expensive, it really does set the new bar for a capability of a model.
00:05:03:10 - 00:05:27:10
AJ Kuftic
And we've mentioned a few of these reasoning models. Now what are those? I think this is something that a lot of people kind of go, oh, it's the reason it's the it's the new model, but it's not the same thing as a I guess we could call it traditional. It feels weird to say, but when we start talking about this, you know, reasoning model, you have the, the O three models, you have clods three and seven thinking all of these different reasoning models.
00:05:27:10 - 00:05:29:23
AJ Kuftic
What is a reasoning model? What makes them different?
00:05:30:01 - 00:05:54:19
Mike Garuccio
So it's at some point the industry realized that you can get a model to perform better if you impromptu and then train it on effectively talking to itself and talking its way through a problem, instead of immediately just trying to give you an answer. So effectively thinking or reasoning model talks to itself or has an internal monologue of what it's actually talking through, and then uses that to give a better response.
00:05:54:21 - 00:06:11:08
Mike Garuccio
The advantage of that is it is able to do some things that a normal model just can't, especially when you start talking about multiple steps in a process. It's able to work through those much, much better. So you'll see that a lot in like coding assistants and a few other places. It's really working well at more complex tasks.
00:06:11:10 - 00:06:34:02
Mike Garuccio
So, go research something, something along those lines. The downside is it's a lot slower because it's doing a whole bunch of internal text generation that you're never seeing or that you don't really care about to arrive at an answer for you. And then as part of that, it's also much, much more expensive, depending on the provider, tokens are either the same price or more expensive than their normal models.
00:06:34:03 - 00:06:49:08
Mike Garuccio
But you're just generating a lot more tokens, so it's going to run your bill up much, much quicker. And so that reasoning models can do a lot of great things, but they really aren't great chat models, and they don't want to be the thing that you're defaulting to in chat.
00:06:49:10 - 00:07:07:21
AJ Kuftic
And I think that's, disheartening for me, as somebody who talks to myself a lot when I'm trying to figure things out, to know that the computers are doing that, too. But I think this is where some of these use cases have really come up around reasoning models. Does it replace the standard sort of use case that most people have of.
00:07:07:21 - 00:07:19:20
AJ Kuftic
Can you help me summarize this thing, or can you give me five questions for something or generate me some bullet points? Does it replace those? Or is this really more for those complex coding type of tasks?
00:07:19:22 - 00:07:38:08
Mike Garuccio
This really is more for those complex tasks. I don't want to kind of pigeonhole it into coding. It is also has a potential in building tools for doing research or much deeper writing, where it's sort of, expecting to be interactive with you and have a much bigger context and kind of longer length of time that it's writing across.
00:07:38:10 - 00:08:01:03
Mike Garuccio
So it's great for specific tasks, but it's just the thing that you want to be intentional about using as opposed to, oh, it's the best thing I want to default to it. You know, you probably want a normal chat model for at this point in like 80% of things. With that reasoning model being available when you do need it for specific tasks.
00:08:01:05 - 00:08:22:12
AJ Kuftic
Mike, this is a really, really helpful information. I want to thank you for your time. And we will, be back after these brief messages. Everyone is interested in going into AI world. Everybody wants to move into the space. But the biggest problem is, how do I do that in a secure manner? How do I protect my company's data?
00:08:22:16 - 00:08:33:01
Brad Reynolds
It's kind of like the table stakes of AI. If you're going to start building applications with AI, you need to make sure that the right people are having access to the right AI models and the right data.
00:08:33:02 - 00:08:40:13
Tom Cooper
We've built our systems to integrate with the systems that you currently have in place for access and controls all throughout your organization.
00:08:40:13 - 00:08:56:00
AJ Kuftic
For AI security, compliance tooling allows you to connect into your existing Active Directory. You talk to one log in Azure, read your single sign on platforms to be able to utilize your existing user directly. Don't have to manage a bunch of different accounts across different platforms.
00:08:56:02 - 00:09:11:22
Tom Cooper
We integrate with the systems you use today, so the implementation is easy and security is guaranteed.
00:09:12:00 - 00:09:16:21
AJ Kuftic
Welcome back. And with me today is Tom Cooper, our product owner for AI. Tom, how are you doing today?
00:09:16:23 - 00:09:18:04
Tom Cooper
Great. How about yourself AJ.
00:09:18:06 - 00:09:37:01
AJ Kuftic
I'm doing great. And we just heard from Mike about Deep Seek and a lot of the challenges and changes that are coming there. And it kind of got me thinking about the improvements to models that we've seen. Right? We saw cloud go from three 5 to 3 seven. We've seen, GPT and OpenAI go from GPT four to GPT 4 or 5.
00:09:37:03 - 00:09:45:06
AJ Kuftic
If you're an organization and you want to take advantage of these changes, like how do you even do that? Or where does that even fit into your overall AI strategy?
00:09:45:08 - 00:10:05:06
Tom Cooper
Yeah. So, I mean, this is one of the reasons that we built the platform that we built, right? So the idea is as these models come out, traditionally, if you've picked one vendor, let's say copilot, right? Copilot is built into a single model. Currently that's GPT four. But you're not going to see 4.5 and copilot for a long period of time.
00:10:05:06 - 00:10:36:08
Tom Cooper
And that period of time is all time, where you can see efficiencies immediately from upgrading to the newer model. Our developers immediately the day they announced cloud three seven, they all wanted access to it. You are one of the ones who reached out to me right away when Gemini came out and said, Tom, Gemini two now, right. So providing people a platform where they can start consuming these, they're already eager to jump forward because the news is keeping them in the cycle and we just provide them the mechanism to do that.
00:10:36:10 - 00:11:00:10
AJ Kuftic
And I think this is where our platform comes into play, because being able to take advantage of multiple models isn't something you can do with every platform, right. We offer the ability to connect to Gemini and Claude and GPT and other models. So how does that play into our overall capabilities and how does that play into somebody's overall strategy?
00:11:00:12 - 00:11:23:09
Tom Cooper
Yeah. So initially that the the direction we would take from the very top is vendor lock in is a problem everybody suffers from if they're trying to pick a model. Right. And it's it's been reference to me as paralysis by FOMO. Like I'm so fearful at missing what comes out maybe next month. You know, deep was a really good, it's a really good example.
00:11:23:09 - 00:11:43:00
Tom Cooper
When Deep Sea came out, the entire industry said, whoa, whoa, whoa, do we have to change? Do we have to move off of OpenAI? When you're stuck into a contract with an individual provider, that's just not something you have the ability to do without waiting for your contract to expire, or finding a way to maybe implement additional cost to bring them in?
00:11:43:02 - 00:12:05:21
Tom Cooper
Our platform, was designed to eliminate, vendor lock in from the beginning. Like any model that can meet our kind of level of, privacy and security concerns, which should be the level that an enterprise needs to use a model the moment they meet that the moment we know they're not training on data, we keep all of those models up to date immediately, so companies actually don't have to worry about it.
00:12:06:03 - 00:12:08:09
Tom Cooper
Their users can just start using them.
00:12:08:11 - 00:12:29:21
AJ Kuftic
And I you kind of touched on that a little bit. I think this is where the the real operational challenges of AI are starting to hit some organizations where they want to take advantage of these things, but there's a compliance angle to it. There's a security angle to it. So how are we ensuring model compliance. Because like deep seek is, you know, a Chinese based model.
00:12:29:21 - 00:12:39:23
AJ Kuftic
So there's a lot of organizations like I don't want to send my data to China. So how are we ensuring model compliance even outside of the you know that specific example?
00:12:40:01 - 00:12:57:22
Tom Cooper
Yeah. So I mean, if we want to go into the details, what we've done, what our compliance team has helped us do. So we couldn't have done this without our compliance team, which is a world class compliance team without their ability to kind of guide us. First, we make sure that all of our vendors we work with are Soc2 compliant.
00:12:57:22 - 00:13:23:22
Tom Cooper
While that's standard, you know, making sure that there's no broken bridge ladders or Remediations that aren't taking care of all of those things that make a SoC report solid, our compliance team monitors those. But then also do they have public attestations and private attestations and the terms of service that are contractual, that help us pin them down to not training on data and not storing data?
00:13:24:00 - 00:13:48:11
Tom Cooper
And the problem with, you know, something like deep Sky, if you're talking about their models from overseas, right, from, from China, their, their soc2 compliance is a different set of compliance standards that we have. So we would never bring that in and make it available to a customer. But if a United States based company who has the same compliance mechanism, let's talk about, you know, Azure Foundry for a second, right.
00:13:48:15 - 00:14:05:18
Tom Cooper
They can bring up DPC if a company really wants to be able to implement that in a secure and safe way, we can open that door for them. We can't remove the biases that might be in the training data, but we could definitely provide that in a secure mechanism for them. I would suggest stay on cloud three seven.
00:14:05:18 - 00:14:07:09
Tom Cooper
It's it's magical.
00:14:07:11 - 00:14:30:02
AJ Kuftic
And this is where I think that ability to pick the model you want to use. You just mentioned cloud three seven being your personal favorite. Some people really like GPT. For some people really like the llama model. Some people want to be able to use something like Deep Sea. How is this beneficial for developers? Because I know that there's always the story of, you know, you can go in and put in whatever prompt you want to and pick your prompts and those sorts of things.
00:14:30:02 - 00:14:45:00
AJ Kuftic
But if you're a developer trying to add AI into your software, into your workflows, how does this and how does our gateway help with allowing those developers to take advantage of those models as they're available?
00:14:45:02 - 00:15:07:13
Tom Cooper
Well, first off, from the beginning, the idea of the gateway was designing it so our our own developers could consume it. So we, you know, rather than just following a specific specification. We started with the most used specification, which is the open AI SDK. And then we wrote the entire translation layer underneath so that any developer that's using us can take either code.
00:15:07:13 - 00:15:28:22
Tom Cooper
They've written to OpenAI before, change the endpoint and change the key. Boom. It works with us. But if they want to take that same code and now make it work with anthropic, they just got to change the model to a cloud model. They don't have to change any other parts of their code. So from the ground up, this was designed to make it easier for developers to build kind of internal tools.
00:15:29:00 - 00:15:39:05
Tom Cooper
And anything that a user sees inside of our chat client know that they could build the exact same functionality. Because our whole chat client is built on our API.
00:15:39:06 - 00:16:00:05
AJ Kuftic
So when we start talking about all of these different models, what is the benefit of having multiple models available to you? Because I think that's something that we're not necessarily at a point where things have settled down. The industry is still innovating and new models are coming out. How are we how do multiple models benefit an organization?
00:16:00:07 - 00:16:32:03
Tom Cooper
So this is a very interesting question as we're moving forward through the development of lives. Remember enterprise use of these started in 2022. Right. These are these are really really new. As I look at let's say we talked about Claude a lot already, right? Claude is the most developer centric model. It's been post-training, which means they finished their model and then they trained it even further on how to develop and how to handle kind of writing unit tests and how developers work.
00:16:32:05 - 00:17:05:15
Tom Cooper
If you've picked a single model, you, you, you chose Copilot or you're with OpenAI and you're providing, let's say, your developers with only those tools where you're really asking your developers to code, write code with something that could be, you know, 40, 50% less efficient and less effective at writing code. If you're asking me to take a large amount of data and when I say a large amount of data, let's say you have 3 or 4 books worth of cases, case files, and you want to feed that to the yellow line to get an information, some information back.
00:17:05:20 - 00:17:37:03
Tom Cooper
You can't do that unless you have one that has a large context window, which would be Gemini. Right. And as each of these kind of use cases unfold, 90% of employees are going to use the default ChatGPT answer my question. It's great. That works really good for me. But when somebody gets to a point where they need a model, giving them access to that model without having to worry about the security concerns, without having to put them through your own internal validation process, without worrying about whether or not that what they're auditing is and whether or not you have the like.
00:17:37:03 - 00:17:49:03
Tom Cooper
All of those things play a huge part of being able to say, do what you do with AI the best that you can. We'll give you the tools. And that's just what we're helping customers say to their to their employees.
00:17:49:05 - 00:18:08:09
AJ Kuftic
And I think that plays into I give access to all of these models with all of the observability and security tooling around it, knowing that those individual models are compliant. And I, as a developer, get to pick which one is the best for me and not have to worry about, okay, well, we're locked in with this one particular model.
00:18:08:09 - 00:18:24:01
AJ Kuftic
We have to deal with this one, even though I know this one over here is better, I can I'm stuck with this one. You can now take advantage of those things. And I think that's really, really powerful. Tom, I want to thank you for your time. I know you're super busy, and we will see you all in the Q&A.
00:18:24:03 - 00:18:34:16
Tom Cooper
Because.
00:18:34:18 - 00:18:52:14
AJ Kuftic
Welcome back, everyone. And joining me now, live from beautiful sunny Cleveland, Ohio, is, Mike Riggio and Tom Cooper. They're in the same room. I wasn't expecting that. I was honestly expecting both of them. They said, hey, we're in the same room before we got on, so it was very nice. It's good to see you guys.
00:18:52:16 - 00:18:53:18
Tom Cooper
00:18:53:20 - 00:19:14:12
AJ Kuftic
We got some really great Q&A. From the pre-registration form. If anybody does have a question again up here, submit a question, scan a QR code. And you can ask your question live as well. The first question was, and I think, Mike, you kind of touched on this was can I reasonably run deep on my typical server infrastructure without the GPUs or specialized components?
00:19:14:12 - 00:19:17:18
AJ Kuftic
And you mentioned, yeah, you kind of need some GPUs to run it.
00:19:17:20 - 00:19:35:04
Mike Garuccio
Yeah. Yeah. You're right for the most part. Do I will modify that a little bit that there are very small versions of the deep spec model. Like there are Lama and many other things that you can run on, like a MacBook. It takes a little bit of work. You still want something with the GPU? Typically, but you can get it to run moderately well.
00:19:35:06 - 00:19:37:17
Mike Garuccio
Those are pretty limited functionality, though.
00:19:37:19 - 00:19:56:07
AJ Kuftic
Yeah, I think that's that's the part here is that the GPUs are what make this really scale and actually perform well. You can run a lot of models without it, but it's not going to be as nice. The next question, I think this this is our sort of main story here is having extensive background working with PII data.
00:19:56:07 - 00:20:19:06
AJ Kuftic
I'm curious to know the best means for incorporating AI functionality while ensuring data security. Are there ways to use AI to protect intellectual property and stay HIPAA compliant and I know that this is Tom. This is something that you and the team have worked on specifically. Can you kind of walk through what that means from a data security and how connecting models into the data is done in a secure fashion?
00:20:19:08 - 00:20:40:01
Tom Cooper
Sure. So I mean, obviously, like we talked a little earlier, making sure that privacy is out front and the data is not being trained on is imperative. But really, when it comes down to HIPAA compliance, understanding your AI governance policy and how people are adhering to that policy, the best tool that we provide is our enterprise observability.
00:20:40:05 - 00:21:02:13
Tom Cooper
It's just the ability to go in and enforce with your user base whatever governing policies you have, whatever regulatory bodies you're responsible for adhering to, not only do we provide the ability to connect to those data sources and pull that data in, but we give you the ability as an organization to go in and make sure that your employees are adhering to those policies.
00:21:02:15 - 00:21:21:00
AJ Kuftic
And this is a this is the biggest challenge with a lot of organizations are facing right now. We want to take on AI capabilities. You want to take on the AI models, you want to connect it to your data. But that data is very important. It's very there's usually personally identifiable information, whether that's H.R data or customer data.
00:21:21:00 - 00:21:38:03
AJ Kuftic
Somewhere along the line there's identifiable information. So being able to do that and what we've built in order to do that is, is actually really impressive. And this is a deep kudos to the entire AI team that we have here, who've done a fantastic job. And who will listen to me say, I want Gemini to now.
00:21:38:05 - 00:22:09:02
AJ Kuftic
So thank you guys. The next one is a is an interesting one because because we're in the infrastructure space, we don't see a lot in the the customer experience space. But this is where we've seen a lot of things pop up, around AI and use cases there because of the nature of that, of that space. So how can I help in those use cases where customers have multiple files of data and they want to be able to use Rag or retrieve log generation to access that?
00:22:09:04 - 00:22:30:12
Mike Garuccio
Yeah. So there's kind of two pieces there. And the first piece and the one that most obviously AI is looking at eight, ten documents and writing a summary of what's in them. The and that's something that just about any of them modern models can do, as long as they have a big enough context window. The other half of that, though, is going and finding those 8 or 10 documents that are actually relevant to the thing that you're asking about and getting those back to the model.
00:22:30:17 - 00:22:49:10
Mike Garuccio
And that's really where having your data in, a searchable format. So a vector store that's also able to do things like, query based searches for titles, things from a specific date, that sort of thing, really comes into play because it gives the model the ability to look up that information and pull it back for you, so that you don't have to go find it before it out.
00:22:49:11 - 00:22:53:20
Mike Garuccio
You ask you to summarize, you can just say, hey, summarize this, and it can go look up the data for you.
00:22:53:21 - 00:23:32:06
Tom Cooper
An integrating API access to a workflow tool dramatically increases that effectiveness. To write internal we we use retool, but we have access to all the vectorized data through the API. We have the ability to pull kind of semantic search from multiple different repositories. And take each of those steps into different steps of analysis, summarization. It really becomes powerful when you tie not just rag and chat, but you tie rag chat and now external tooling through, you know, what is, an API that grabs basically every single model we have and every data source.
00:23:32:06 - 00:23:34:11
Tom Cooper
It's incredibly useful.
00:23:34:13 - 00:23:54:03
AJ Kuftic
And this is, I think, where it's starting to meet the road. And the question here is very good because it's actually getting to the base question that I always ask when someone is saying like, how do I bring AI into my environment? I always ask why? What is the thing you're trying to solve? What is the use case you're trying to hit?
00:23:54:05 - 00:24:13:06
AJ Kuftic
Because otherwise it's just generic tooling. It's not really doing anything of note. When you have a specific use case, you're trying to solve for it. That's where you actually get value. That's where you could say, okay, this is needs to be compliant. This is how why I need to connect it to my internal data versus just here's Copilot or here's, you know, ChatGPT.
00:24:13:08 - 00:24:39:00
AJ Kuftic
We can go to the next question. And this one I'll kind of take, a good start at here. As data center operators, what changes are you seeing in power cooling space requirements with the rise of reasoning models in high density AI workloads? A few weeks ago, I was actually at a data center summit. There was more or less AI's causing a lot of power needs, and we need to build more data centers in order to do this.
00:24:39:02 - 00:25:01:10
AJ Kuftic
It's a very, very interesting topic to go through because right now we're kind of struggling with data center power already. Northern Virginia is running out of space. Central Ohio is running out of power in space there as well over the next few years. And AI is not going to make that any better. What most people will see in their data center is somewhere between 8 and 10kW per rack.
00:25:01:12 - 00:25:23:23
AJ Kuftic
This is a fairly common amount of power being driven to Iraq for virtualized workloads or server workloads, storage arrays, those sorts of things. The when you bring in a AI platform, we have seen 50kW per rack or 100kW per rack. And while that requires a ton of power to be driven to the rack, that also means you have to.
00:25:23:23 - 00:25:25:09
Tom Cooper
Cool all of that.
00:25:25:11 - 00:25:44:17
AJ Kuftic
And a lot of data centers need to have changes to their cooling lake. Forced air doesn't work anymore at that. At that power draw force, there is not enough. So being able to have things like liquid cooling and those sorts of things, are, are things that you actually have to bring in. And that is a big investment for a lot of data centers.
00:25:44:17 - 00:25:59:18
AJ Kuftic
That's why you're seeing these effectively new data centers being built with this specifically in mind to take advantage of that. And Tom or Mike, I don't know if you guys have seen, or heard anything in your conversations around data center usage for AI.
00:25:59:20 - 00:26:19:19
Mike Garuccio
So I largely haven't I think a majority of our customers are really going and consuming public models and consuming things that are sort of offered as a service from elsewhere. We don't really have a base of customers that are looking to do that high density, really GPU driven workloads. It's a thing that we can support, but it's not a thing that we really target.
00:26:19:21 - 00:26:34:05
Mike Garuccio
And so a lot of that ends up happening at other facilities. Certainly. Do, you know, keep an eye on the new designs coming out from Nvidia and see, you know, these hundred kilowatt racks coming through that I don't envy the people that need to figure out how to support.
00:26:34:07 - 00:26:56:01
AJ Kuftic
Yeah, this is actually I think one of the big things that people got excited about with Deep Seek was if you could actually run and train AI models and use a fraction of the power and the cooling, that means that, yes, that has an implication to Nvidia stock and other companies who are in that space, but also has an implication to where you can run these workloads.
00:26:56:03 - 00:27:12:15
AJ Kuftic
And we've also seen a number of questions around AI at the edge, for things like inference and machine learning, there too. So being able to have GPUs in remote locations can be interesting. It's just going to be needed. It needs to be targeted to specific use cases and not I want to just run the biggest models possible.
00:27:12:15 - 00:27:14:22
AJ Kuftic
And I think that's kind of where this comes back to.
00:27:15:00 - 00:27:38:22
Mike Garuccio
Yeah, I agree there. And I also would say that there's I haven't seen traditional server infrastructure and small GPUs. I think there's a lot of potential in small ish models running on like Nvidia GPUs at the edge or in the data center. Outside of the really large models, they can kind of serve a different purpose.
00:27:39:00 - 00:28:00:01
AJ Kuftic
All right. Let's bring in, the next question here. So how and when will I have an effect on the daily tasks of my average users? I do want to thank, someone for asking this question because I think this is very, interesting because the answer is probably it is. And you don't know it yet.
00:28:00:03 - 00:28:24:12
Tom Cooper
So this question is asked on almost every, sales call. It's probably asked again twice in every onboarding call. Right. So, I'll give you the fun answer and then we'll give you the real answer. Right. The fun answer is if you just give Gen AI to people today who, use digital communication, you know, the average is 5 hours to 10 hours a week.
00:28:24:12 - 00:28:50:14
Tom Cooper
People are spending drafting emails, drafting. This is not reading emails. This is not the three of our inboxes. You know, deleting emails are sorting them. This is writing emails. And if you spend 5 to 10 hours a week writing email and that that alone you gain 50%, 80% efficiency on what you're talking about. You know, releasing anywhere from 3 to 8 hours of 40 hours every week.
00:28:50:14 - 00:29:13:00
Tom Cooper
Back to somebody's calendar. I think there's efficiency gained right there. I don't think you need to look any further to understand how big of an impact that can have on an organization. If you add accuracy, you accelerate the rate at, you can run it, you add efficiency, and then you even add a tinge of professionalism and almost kind of brand control that you can't get any other way.
00:29:13:04 - 00:29:39:19
Tom Cooper
So I think those exist immediately. But the moment you attach rag to it, right. So attach your own internal documents before we even talk about workflow changes. Those are all big. We can talk about what those look like for hours. Things that change your business. If I can just help people find, information about their benefits quicker, find their training guides quicker, find their sales material before a call quicker.
00:29:39:19 - 00:30:04:06
Tom Cooper
Every one of these things are incredibly easy to, to obtain with our system. And they're so low hanging fruit that it's almost impossible to to not do them. And in doing that, you drive an amount of efficiency on both sides of the ball. That's almost incalculable right now, we legitimately tell people you're ROI positive before you come out of the three month free trial that you saw there.
00:30:04:12 - 00:30:20:03
Tom Cooper
There's no other tool and technology can do that. That's how effective AI is. And it's day one. The moment you give somebody, log in to the chat. My estimation, if you tell them, start writing your emails with this, you immediately are almost ROI positive right there.
00:30:20:05 - 00:30:39:14
AJ Kuftic
Yeah, I think the one that I see a lot, the one the use case I usually point to a lot is HR onboarding or even just HR general questions, because everybody has the four. There's like four pieces of information you actually need, and you don't even need to get into actual PII data of I'm AJ and this is my address, Social security number and all that stuff.
00:30:39:14 - 00:31:08:10
AJ Kuftic
It's you have a an employee handbook, a holiday schedule, a pay day schedule and a benefits probably a PDF that says, here's all the health insurance plans that our company offers, and you can answer probably 80% of the questions that most people have because nobody wants to read the documentation. They'll chat with it all day long now. And so this is where when you add rag sources with your companies, maybe that's internal documentation procedures, whatever.
00:31:08:12 - 00:31:23:19
AJ Kuftic
And it can ask questions and link back to the original document. That's massive because it's getting people to the answers they want faster without having to put like super secret, identifiable information at risk.
00:31:23:21 - 00:31:50:13
Tom Cooper
I can ask, tell me about our PPO plans to our chat faster than I can open up outlook and find the right person to put in the email box. I know that sounds ridiculous, but it's absolutely true. The efficiency that it drives for me is incredible. But then when you think about the efficiency of the H.R department, our own internal H.R department, we could see the excitement and the clapping when we told them we were coming out live with this.
00:31:50:15 - 00:31:51:07
Tom Cooper
Yeah.
00:31:51:09 - 00:32:09:10
AJ Kuftic
Hi, Molly. They were thrilled because they got to see it. We put it in place for them, and it was really. Here's a SharePoint folder with these four documents in it. Yep. And they immediately got to see like oh, or you could just ask those questions and it just answered oh this is great. Thank you. Yes. Please roll this out right now.
00:32:09:12 - 00:32:14:15
AJ Kuftic
And it didn't really put anything super critical at risk. And it actually saved them a ton of time.
00:32:14:19 - 00:32:16:01
Tom Cooper
Yeah.
00:32:16:02 - 00:32:45:02
AJ Kuftic
Let's go to the next question here. What compliance controls need to be looked at before onboarding an AI tool? How do you protect from data loss of intellectual property? And we kind of talked about this earlier in terms of, you know, our security compliance team and what they're doing. But Tom, kind of or maybe Mike walk through what they're really looking for or when we see a new model, what are you looking for to bring to our compliance team to validate and make sure that everything's good?
00:32:45:02 - 00:32:48:15
AJ Kuftic
And when we add a model, we're not putting things at risk.
00:32:48:17 - 00:33:14:18
Mike Garuccio
Yeah. So the number one thing that we're looking for is a, you know, up to date and valid soc2 report. We can then give to our compliance teams and have them go through line by line and really validate that everything there is up to the standards that you'd expect. It had some surprises with vendors that are fairly widely used that once we actually started doing some scrutiny on their SoC TOS aren't really valid, missing bridge letters or other things like that.
00:33:14:18 - 00:33:25:16
Mike Garuccio
So it's it's important to actually dive in and have your compliance expert go through line by line of their report and make sure that there's not a hey, we just decided not to do this in there because you'll find a few of those.
00:33:25:16 - 00:33:51:05
AJ Kuftic
Sometimes I think that's that's kind of where this comes back to like almost the rubber meeting the road of, yeah, all this AI stuff is really, really cool, but nothing about what we do in it. And from a compliance and security standpoint has really changed. Right. So we don't have a an ability to just ignore that we have to continue doing this.
00:33:51:05 - 00:34:15:06
AJ Kuftic
And so going through, compliance regimes like this while increasing. Lee. Not interesting. I have a master's degree in it too. And I'm telling you, it's not that interesting. It is something that we have to do for the various compliance legal things, because if you don't and you leak data, you're liable for legal action, and that becomes its own, a larger problem there.
00:34:15:08 - 00:34:37:19
Tom Cooper
Not this. On the same note, when you look at just sass in general, the the modern version of data protection and sass is the DPA, right? That data privacy or data protection attestations, we take that same level where we're looking. And we talked about this a little bit earlier in the questions. We take that Soc2 report and with that being table stakes.
00:34:37:21 - 00:35:04:21
Tom Cooper
And then we add into it that not only does, vendor have a DPA that lays out that they will not train or store data without, without express written consent. So you get some understanding from the public side that they have pressure that also is referenced in all of their terms of service when we engage with them to make sure that they're contractually held to that.
00:35:05:03 - 00:35:22:14
Tom Cooper
So you start with SoC two compliance and then you layer on top of that the attestations for data protection and data privacy, that they won't train on that data. And you really do end up with a solid set of guardrails to be able to trust pushing your data through one of these public models.
00:35:22:16 - 00:35:49:10
AJ Kuftic
Yeah. And that's actually something that doesn't get talked about enough in terms of the training piece. When you when your users are using the free ChatGPT options, which they probably are, the free model automatic trains on data, that is the default anyth anything that you put into the free versions of ChatGPT end up in their training data, and then can come out in a response somewhere else in the paid versions, it depends on the level.
00:35:49:10 - 00:36:14:07
AJ Kuftic
So you kind of have to look at their, there's a document that they have, and if you're on the free one and you don't want it to train on your data, you got to dig through the settings to find that. But it's not easy to find and it's meant that way. So these are the things that we see a lot of CIOs and CISOs when they talk to us about, hey, I the very first use case, honestly, that most people come to us with is, hey, I know my people are using ChatGPT, can I give them ChatGPT?
00:36:14:12 - 00:36:26:23
AJ Kuftic
But I know what's going on and that's where observability and security and compliance pieces come in. So I think that's really just the simplest place to start, is giving people access to the tools they like to use, but making sure that everything's protected.
00:36:27:01 - 00:36:27:06
Tom Cooper
Yep.
00:36:27:07 - 00:36:51:20
Mike Garuccio
Yeah. And I would also add one last item on compliance there. Sorry. It's about more than just keeping your data secure and making sure it's not going to leak or using the training set. It's also about being able to prove to auditors, regulators, anyone else what your people have and have not used AI for, particularly in the healthcare industry, there's a lot of questions that are there around whether it's being used in a patient context, which it generally is not allowed to be.
00:36:51:22 - 00:37:06:14
Mike Garuccio
Proving that it's not is a whole lot easier if you have a audit log of everything your team has ever used the AI tool for, and, you know, they have never gotten at least about significantly working to bypass your controls, haven't gotten to any other AI tooling.
00:37:06:16 - 00:37:29:04
AJ Kuftic
Yep. So we'll take, I think there's 1 or 2 more questions left. What are business reasons for and against using AI solutions that are integrated with SaaS platforms? And, Tom, we talked about Copilot. Obviously Salesforce has one. It's like if you can name a platform, somebody has an AI tool inside of it. Matthew McConaughey sitting in the rain for Salesforce for it.
00:37:29:04 - 00:37:41:08
AJ Kuftic
So this is the sort of, thing that we run into a lot of, well, which one do I use? Which one or which ones are worth it? Which ones aren't? And that's not necessarily the question to be asking there.
00:37:41:10 - 00:38:09:11
Tom Cooper
The real question to ask here is are what is the level of assurance I have and comfortability that I have with using those embedded and built tools and understanding whether or not they're secure and whether or not their usage is compliant. There are lots of tools on the market that, are not our tools that, are are not only beneficial to an organization, but should be the zoom call recordings.
00:38:09:11 - 00:38:39:13
Tom Cooper
It's a very good one, right? It dramatically assists in our day to day process. Same with teams call recordings. I wouldn't want to turn that feature off for any feature. Now you can use fireflies or other tools where you can plug into a different model on the back and get some of the same attestations. So the real answer comes down to if the tool is providing, if the flashy eye button in the tool is providing a benefit that is calculable and demonstrable, then use that tool.
00:38:39:17 - 00:39:08:11
Tom Cooper
As long as you're comfortable with its level of security. But in almost every case, in reality, the reason why, when polled, 92% of CIOs are implementing AI right now, but 30% of CEOs, only 30% think there's any value today is because all of those tools are so wide that actually getting benefits from using them in the flow doesn't help your business in what you do directly.
00:39:08:13 - 00:39:33:07
Tom Cooper
So really, it comes down to calculate the risk versus the value that you're getting back from that tool. And if you can't calculate the value from that tool, if you can't separate its usage, if you can't understand how it's impacting the business, then it's probably a better solution to take something like ours and build a workflow where you have total insight and direction into every step that's taken, every piece of data that's used.
00:39:33:13 - 00:39:48:12
Tom Cooper
And then you can actually calculate what the return on investment are and end those solutions and what the efficacy rate of AI is for each one of those workflows. So I know I should say only use expedient, but there are great tools that aren't expedient. Mike.
00:39:48:14 - 00:40:08:02
Mike Garuccio
Yeah, I think it's really about, you know, like you said, is does that tool have some unique value that being embedded gives you? There's undeniable reasons that zoom, you know, using their AI companion, is worth doing over trying to incorporate it with some other model. And there's other things that are similar. You know, if your entire life is in Salesforce, it may make sense to you.
00:40:08:02 - 00:40:29:21
Mike Garuccio
Salesforce is AI tool because it's already there. The thing is, not a lot of companies or businesses have one tool that there are people working all day, every day, and there's a lot of advantage if you can take step out of that individual tool and have one place that they can go that's talking to Salesforce, but also knows what their outlook, emails and calendar looks like, and also has information from their SharePoint drives.
00:40:29:21 - 00:40:34:19
Mike Garuccio
And it can use all of that together and give one better response.
00:40:34:21 - 00:40:56:14
AJ Kuftic
Yeah, I think that's the I think the big one is, is does the platform itself having access to that data, or does the platform itself itself being able to collect some form of data like a zoom call recording? There's other tools that you can integrate there, but the zoom one being built in means I click the record button or I use their AI companion.
00:40:56:14 - 00:41:17:01
AJ Kuftic
It's just baked in. I don't have to do get okay, record the call, then get the transcript, then take the transcript and run it through a different tool. It's there's a native flow there. I think that's really where that value comes into play, and that's where the differentiation happens. And then in other places where it's like, I just need to get the data out and that I want to do something with it.
00:41:17:01 - 00:41:40:16
AJ Kuftic
That's where our tool definitely comes in. Speaking of our tool, how is the Experian AI platform priced in other in other words, how can I develop a total cost of ownership model and just from a base level, our AI platform is based on company size. So instead of saying this many tokens per user or this many individual users, it's based on, t shirt sizing in terms of company size.
00:41:40:16 - 00:41:47:20
AJ Kuftic
And Tom, you can feel free to go really explain all of this out in terms of total cost of ownership, because you do this all day.
00:41:47:22 - 00:42:20:13
Tom Cooper
I think the the most appropriate answer for this audience is come talk to me. I can find the size that works best for you. We don't charge per head what we try to do. Since you can consider, the solution that we provide as almost, private cloud itself, where we build your own infrastructure for you, we manage that infrastructure in order to ensure the security, but that also enables us to provide the chat client, which saves you that cost, that you would traditionally have to pay.
00:42:20:15 - 00:42:42:06
Tom Cooper
Across the board for per head. See, licensing. And that comes at a dramatic savings for organizations. So, talk to your your ag talk to your local ramped up to your connect. Hop on the phone with us. We can happily show you how the pricing works. And, you know, usually when people are not forward with pricing, it's because the pricing is high.
00:42:42:11 - 00:43:04:19
Tom Cooper
We're actually in a position where we're driving anywhere from 50% to, you know, 80% savings over, a single tool, right? Like like copilot, like, ChatGPT. While providing access to all the models. So that's enough to get you excited. Please feel free to hop on a call out how detailed it all for you.
00:43:04:23 - 00:43:20:10
AJ Kuftic
Yeah. And, once, once we wrap up, there will be a QR code right here on the screen and probably down here at the bottom that you can scan to set up time. With us, we'll bring in Tom to talk through those things. So, stick around for a little bit. Right after we're done, you'll be able to get to that code.
00:43:20:12 - 00:44:03:12
AJ Kuftic
Last question here. And this one was kind of interesting. This one came in live, given the current power implementation limitations. Excuse me, does the need to support AI workloads outweigh the benefits of maximizing spatial density within a single data center, specifically with things like security cages? Yeah. You're going to find this a lot, in a lot of places, security cages and those functions, trying to limit the physical space because you're trying to stick inside of a specific number of racks or you're paying for floor space, a lot of data centers in order to hit the power needs, not necessarily the density or the cooling purposes will spread that workload
00:44:03:12 - 00:44:29:06
AJ Kuftic
out. Because everybody wants these like 50 kW racks when previously it was like 8 to 10. We've heard of 160 kW racks, which is bananas to me. That's five kW per unit, per rack unit. Meanwhile, a few years, like less than ten years ago, it was five kW per rack. So that's wild. All of this will come into play, especially when you're talking to data center providers, including us.
00:44:29:06 - 00:44:47:15
AJ Kuftic
Honestly, where spreading out those workloads helps with cooling. It helps with being able to hit the power needs that you have. But at the same time figuring out ways to secure that. If you need to have a cage, do you really need to have a cage, or can you just lock the rack itself? There's different ways to go about that.
00:44:47:17 - 00:44:54:12
AJ Kuftic
And if you have questions about that, we can definitely talk. Mike. Tom, any last thoughts before we wrap up here?
00:44:54:14 - 00:45:10:08
Mike Garuccio
I just kind of added that that, oddly, it makes security cages almost free. Because you're going to be using up all that floor space anyway. It's, you know, normally you'd be paying for that square footage, but you're paying for that square footage if you're going to go, you know, three data centers, typical rack power density.
00:45:10:08 - 00:45:14:22
Mike Garuccio
Anyway, so you're basically paying for some fence.
00:45:15:00 - 00:45:16:23
AJ Kuftic
Yeah, I think it's got guys,
00:45:17:01 - 00:45:19:21
Tom Cooper
Nano separators. If you had to comment on that, go ahead.
00:45:19:23 - 00:45:50:07
AJ Kuftic
Yeah. I think the security cage mentality, is something that we see a lot with specific regulatory requirements, that sort of thing of I have racks, those are secure, but I need to secure my whole physical footprint. And I think there's definitely some value still in there, but it will definitely change the way that you are going about looking at your Rackspace when you need to have that level of power density, and how does the provider actually give you that space in that power without ruining everything for everybody?
00:45:50:09 - 00:46:13:20
Tom Cooper
And I think one little piece of information, just to take away from this, we talked about two different sides of AI here. We talked about, putting up privacy and compliance around the public model usage because of how advanced they are and how much investment they've made there. It makes a lot of sense at the cost savings, to use the public models when they can.
00:46:13:22 - 00:46:35:20
Tom Cooper
And on the infrastructure side, we're talking about the cost in order to kind of spin up your own private models and the world between that exists, and that's what we're trying to help people navigate. So on a scale of I'm okay with public models, but make them private all the way to I need an edge device that's air gapped sitting inside of this location.
00:46:35:22 - 00:47:01:20
Tom Cooper
Every everything along that spectrum, we're able to service from an AI perspective. So really you can set, you know, at what risk level are you at. And then at what budget level are you willing to commit. And then all of those cases, we're able to solve problems. And you know, we probably talk about these too much separately, but this is a good segue in the end, to understand that it's the entire spectrum that will net we're able to help you service.
00:47:01:22 - 00:47:20:05
AJ Kuftic
Yeah. And I think that's a really, really great place to end. If you want a three month free trial and you get to talk to Tom a whole lot, go ahead and scan this QR code down here. This is a risk free trial for three months. We bring in the gateway that we've talked about earlier.
00:47:20:05 - 00:47:42:19
AJ Kuftic
You get access to the models. We also set up a data connector and some storage. So you can do things like that. H.R. Onboarding use case, trying to help you get value out of it quickly. And that's really where this comes in. We see a huge ROI. I mean, Tom mentioned its ROI positive from day one, but this is where we start to see, like the use cases and the light bulbs go off and people go, oh wait, I could do this too.
00:47:43:00 - 00:48:01:12
AJ Kuftic
Or what about this? So go ahead and scan down here on the QR code. We're going to leave that up, after we wrap here. Tom, Mike, thank you so much. For the time earlier in the conversations and then coming back, for Q&A time. I really appreciate it. You guys are great. And audience, thank you for joining us.
00:48:01:14 - 00:48:18:03
AJ Kuftic
Join us next month for something that I have been excited about for the last, like, I don't know, six months. I've been talking about this for quite some time. We mentioned security cages and racks and power, and we're going to go on a data center tour with our senior vice president of data Center engineering and operations, Eddie Zarco.
00:48:18:05 - 00:48:37:02
AJ Kuftic
And we're going to talk through what makes up the back end, the base platform, the physical nature of all of the services that we provide and all the value that we put in, I'm incredibly excited about this. Join us next month. More details coming soon on dates and times. But you're not going to want to miss that one, I promise.
00:48:37:04 - 00:48:38:18
AJ Kuftic
Thanks, everybody. We'll see you next month.