Tom explores the critical decision between building custom LLM models versus using off-the-shelf solutions. Drawing from insights at the AWS Expo, he breaks down the real costs, challenges, and strategic considerations for organizations evaluating domain-specific AI implementations.

Build vs Buy: Making Smart Decisions About Custom LLM Models

Key Topics Covered

When to Build Custom LLM Models

Domain-specific applications requiring specialized knowledge
Handling proprietary or confidential information
Real-world example: AIDoc's experience at AWS Expo
Understanding your organization's unique requirements

True Costs of Building

Data Preparation
- Gathering organizational historical knowledge
- Creating validation and training datasets
- Organizing proprietary information
Training Expenses
- GPU infrastructure costs (billions spent by OpenAI, Anthropic monthly)
- Ongoing computational requirements
- Budget considerations for organizations
Maintenance & Updates
- Keeping pace with base model improvements
- Avoiding being locked into outdated versions
- Continuous investment requirements

When to Buy Off-the-Shelf

Non-hyper-specific use cases
Data collation and comparison tasks
General analysis and processing needs
Cost-effective solutions for standard workflows

Optimizing Model Selection

Using platforms like AWS Bedrock for model diversity
Balancing accuracy vs. cost vs. performance
Example: Claude Opus vs. Sonnet vs. Haiku trade-offs
Avoiding "overkill" with expensive models
Testing and validation strategies

Key Takeaways

Don't default to the most expensive model
Test multiple options before committing
Understand total cost of ownership for custom builds
Match model capabilities to actual requirements
Consider the rapid pace of AI ecosystem changes

Mentioned Companies/Platforms

AWS (Amazon Web Services)
AWS Bedrock
AIDoc
OpenAI
Anthropic (Claude models: Opus, Sonnet, Haiku)

Resources

AWS Expo insights and presentations
Open source foundation models for custom building

Chapters

0:02 - Introduction: The Build vs Buy Debate
0:25 - When Building Custom Models Makes Sense
2:02 - The Real Costs of Building Your Own Model
3:35 - Real-World Example: AIDoc at AWS Expo
4:09 - The Case for Off-the-Shelf Solutions
5:44 - Optimizing Model Selection and Cost
6:46 - Final Recommendations and Wrap-Up

What is The AI Briefing?

The AI Briefing is your 5-minute daily intelligence report on AI in the workplace. Designed for busy corporate leaders, we distill the latest news, emerging agentic tools, and strategic insights into a quick, actionable briefing. No fluff, no jargon overload—just the AI knowledge you need to lead confidently in an automated world.

So today I wanted to have a quick

discussion about sort of build versus

buy when it comes

to different LLM models.

So the reason I bring this up is

because it comes up in discussion like reasonably frequently

and you know there's two thoughts, there's two

schools to this argument.

So if you're trying to do something that

is particularly specific to a certain domain that

you operate in, then taking a foundational model,

a frontier model and then building on top

of it, one of the open source ones

of course, building on top of it to

be able to add more domain specific information

can be a very

effective way of building an LLM supercomputer,

so to speak, that allows your

model to do something that's very domain specific.

Now of course in doing that,

it's going to cost both

time and money in a number of different

ways and so I just want to sort

of touch upon what some of these different

issues might be, especially as you look forward

at the rate of knots that stuff changes

inside of this ecosphere

at the moment.

So if you were to build your own

model, first of all of course what you

need to be able to do is get

a lot of validation training data put

together and come up with a plan to

allow you to be able to actually build

that stuff out.

So you need to be able to get

your entire organisation's worth of historical knowledge,

stick that together and start training

your model on what some of the issues

are, some of the things that you want

to be able to raise with the platform

that you're doing.

Then on top of that, you also need

to start thinking about the

cost of actually training it because of course

training an LLM is not cheap as we

know because OpenAI,

Anthropic and what have you are spending billions

of dollars on a monthly

basis to pay for all the GPUs that

they need to be able to train the model.

So when it comes to actually expanding that model further,

you're going to have to spend a lot

of cash training your model

and of course that might be worthwhile but

this is the type of stuff that you

need to be able to ascertain.

Is it worth the payoff when

it comes to building a dedicated model?

And the third thing of course is that these models,

the underlying models, the open source models that

you're going to build on, improve over time

and so as those models continue to improve,

assuming that you want to be able to

keep up and leverage some functionality that's

in the underlying model that would improve

the performance of your new model,

you have to be able to of course

account for that as well.

You don't want to like,

probably don't want to train a model and

then be stuck on that model version forever.

And so when it comes to build versus buy,

the build side of it can be quite

a costly experience but

I was down at the AWS Expo last

week and watched a talk by

AIDoc and the experience

they had in terms of building a domain

specific model out for the stuff that they

were doing and it made a lot of sense.

If you're looking to do something that is

super specific and it also

involves probably a lot of confidential or proprietary

information that the open source models may not know about,

then building your own model makes a lot of sense.

Now the flip side of that of course

is if you're trying to do something that

is not hyper specific but

requires some degree of thought and

common sense being applied to data that's coming

into your system,

then there's also an element of well just

use what comes off the shelf so if

you weren't doing something that's particularly domain specific

and instead was doing something that was the

collation of data or comparing of data or understanding

something that's going on in your data set,

well then conversely what you can do is

figure out which model works best for what

you're trying to do because for example, if I'm

using Table and the other Sonnet

and Heiku models are not going to be

accurate enough for what I want to be

able to do so I'm probably going to

use Opus and I'm probably not going to

fiddle around too much but if I was

building out a LLM

process to do something like data manipulation

or data comparison or whatever, Opus would A.

be pretty slow, B.

it would cost quite a lot of money and C.

I'm suspecting would be overkill for most things

that I would like need to do and

so if you've got access to Bedrock or

another similar platform

where you've got an array of different models

from different providers to do different things, ensure

that you spend some time figuring out

the accuracy versus cost

and performance trade -offs because they're definitely there

and so if you're spinning up

a process that requires some element of

data validation or whatever and you want to

use an LM to do it, that's cool

but you don't necessarily have to pay for

the most expensive model every time, you might

just get away with

paying for the more

affordable ones, ones that are going to work

more effectively, more efficiently

for what you need them to do.

Don't just go big or go home and

find something in the middle, that's just my

thoughts on it.

So if you want to build your own model,

just make sure that you understand the

what's involved before you start doing that

because you might end up trapping yourself in

the scenario you don't want to be in

and then if you want to be able

to build stuff or buy stuff off the shelf,

ensure that you test and

validate whatever you want to be able to

do with the different models to be able

to find the model that works best what

you want to do.

So that's it, that's my thoughts for the

day, happy Monday,

my name is Tom, I'll be back for

another AI briefing soon enough,

bye for now.