Ned: Other than your mic stand, everything else okay [laugh]
? Chris: Yeah.
As usual, with the things that I observe, this is something
that I figured out, like, years ago but never put a name on.
Ned: Mmm, okay.
Chris: So, let’s start with the obvious.
Coke is clearly better than Pepsi.
It’s not a conversation.
Put your finger back down.
Ned: [laugh] . Okay?
Wha—
Chris: No, no, no.
Don’t you okay me.
Ned: [sputters in the background]
. Chris: Hey.
Ned: Diet Pepsi is better than Diet Coke.
Chris: First of all, they’re both bad.
Ned: Fair.
Chris: The only decent diet soda on Earth is Diet Dr.
Pepper.
There I said it.
Ned: Ohh, I agree that it’s good.
Moving on, what was your actual point?
Chris: I don’t remember.
Ned: [laugh]
. Chris: The other thing about Diet Dr.
Pepper is it’s got two-and-a-half times the
caffeine, so this episode is going to go fast.
Ned: [laugh]
. Hello, alleged human, and welcome to the Chaos Lever podcast.
My name is Ned and I’m definitely not a robot.
I am a real human person who has feelings, dreams,
and an ordinate desire to consume caffeine and sugar.
Like, all the sugar.
Give me all the sugars that you have.
I will suck them out of your blood, thank you very much.
With me is Chris, who is also full of delicious sugar.
Hi, Chris.
Chris: One has to assume that vampires have an opinion.
Ned: An opinion on blood types?
Blood sugar levels?
Chris: Yeah.
So, think about beef, right?
We know there’s a difference between corn-fed and grain-fed, right?
Ned: [laugh] . I believe you mean grass-fed, but moving on.
Chris: Is grass not a grain?
Ned: Oh, I thought you said brain-fed, and I
was like, not aware of any cows eating brains.
Isn’t that how we got the mad cow disease?
Chris: Yes.
Ned: It is.
Okay, so yes, corn-fed versus grain-fed versus
grass-fed, there is a definitive difference.
Chris: Right.
So, if you were eating either human flesh or drinking human
blood—or both I’m not here to shame anyone’s kink—you’re going to
be interested in that person’s, let’s call them the donor’s diet.
Ned: Absolutely.
Chris: That’s why—I’m almost positive this is correct—on the website for the Red
Cross, they say you should do five or six shots of whiskey before you donate,
so that, you know, when somebody gets your blood, it’s a little pick me up.
Ned: [laugh] . Settle down.
I was wondering where you’re going with that.
And it turned out it wasn’t anywhere good.
But hey, speaking of consuming entirely
too much stuff, guess what I did this week?
Chris: Went somewhere that wasn’t good?
Ned: Oh, I went somewhere that was fine, but I consumed
entirely too much of a thing, and that thing was AI.
Wooo.
Chris: Yay.
Ned: Ugh.
So, this past week, I attended the AWS Summit in our
nation’s capitol, and home to a really good taco place.
Seriously, it’s called Rebel Taco.
It’s on the corner of 6th and K.
It’s a less than five-minute walk from the Walter E.
Washington Convention Center.
So, if you happen to be at that convention center: Rebel Taco.
Walk the, like, four blocks, it’s worth it.
I got three different tacos.
They were all delicious, especially the lamb birria.
That was really good.
And the margarita I got to go with my tacos was absolutely not fucking around.
Ohh, somebody had a very heavy hand at Rebel Taco, and I didn’t mind one bit.
Chris: Unrelated, I saw something interesting slash terrifying.
Ned: Okay.
Chris: The new summer drink is a pickled juice-based margarita.
Ned: Thanks, I hate it.
Chris: Anyway, enjoy your weekend.
Ned: So, when I was 15—of course I have a story for
this, right?—when I was 15, I worked at a pizzeria.
And I also had acne because I was 15.
Chris: Right.
It’s like the rule.
Ned: And working in a pizzeria was definitely not going to this.
The amount of grease in the air—Anyway.
So, the owner and proprietor of said pizzeria told me that a cure for
acne—or at least a treatment was drinking pickle juice—and being as
we were a pizzeria that made other things, we had giant pickle jars,
so she just poured me some pickle juice, and she was like down it.
This will help with your acne.
It did not—
Chris: No, no it wouldn’t.
Ned: —but she got to watch me drink pickle
juice, which I think is, like, 99% of it.
Chris: Mm-hm.
Ned: Yeah.
Chris: That was a game of watch the 15-year-old do something dumb.
Ned: Mm-hm.
That seemed to happen a lot.
Chris: [laugh]
. Ned: Sometimes unprompted.
Often unprompted.
Almost always unprompted.
Chris: One day, you’ll stop playing that game.
Ned: [laugh] . When I’m dead, Chris [laugh] . Anyway, so I will not
be drinking this cocktail, but I will be talking about AI for a while.
We’re going to be drinking from the AI firehose, which ironically, is an
AWS service that didn’t come up in all of the presentations that I attended.
For those not familiar with tacos, that’s when you take a corn tortilla,
and you place some toppings inside the tortilla, and then you fold it up—
Chris: [coughs significantly]
Ned: And then—wait, okay, no, wait, we were talking
about the AWS Summit, and not delicious food.
Sorry, I got sidetracked.
It is lunchtime, to be fair.
So, let me talk about the amazing French toast that I had at the Delegate.
It was a Challah-crusted French toast.
Chris: [sigh]
. Ned: The crusting was cornflakes and—stop it, Chris.
All right, fine.
I’m not procrastinating.
You are.
For those not familiar with the AWS Summits, these
are many events put on by AWS all around the globe.
They’re typically one or two days, and it’s kind
of—you can think of it as a speed-run of re:Invent.
And you don’t have to go to Las Vegas.
So, that’s a gigantic win.
I’ve attended the NYC Summit three or four times, and really enjoyed it.
In fact, it’s happening again in a few weeks, and I plan to go.
But I also wanted to try out the DC Summit since I figured it
would have a different focus and attract a slightly different
crowd, by which I mean public sector, government, and military.
It was interesting to see the different types of vendors this
attracted at the Expo, and how the sessions focused on public
sector, and intelligence applications instead of building the
next Uber for Tacos, a company that I would absolutely invest in.
Chris: Yeah.
Let’s talk about that instead of AI.
Ned: [laugh] . We probably could.
But no.
While I was browsing through the sessions, trying to plan
out my schedule for the two days, I couldn’t help but notice
how many of the sessions had AI somewhere in the title.
I actually talked to an AWS person the following day, and they
said that 55% of the titles of the sessions had AI in them.
Even more of them were about AI, but that was
just the ones that had it directly in the title.
Chris: Considering there are 380-something services,
there’s no possible way that 55% of the services need AI.
Ned: [laugh] . No.
Chris: Explain.
Ned: [sigh] . Well, at first, I was just trying to avoid those sessions.
So, I was trying to pick stuff like running EKS clusters, or
AWS networking at the edge, or scaling storage best practices.
You know, things that I think of as, quote, “Real topics.”
Chris: Which, ironically, would include the
responsible and practical use of Firehose.
Ned: Indeed.
Chris: See, I know things.
Ned: I know you do.
Once I finished trying to assemble this schedule, I
looked at it and I realized it was kind of sad and boring.
Like, say, your dad’s old record collection.
Chris: There’s a lot of Boz Scaggs.
Ned: It’s all stuff that’s been around for a decade or two, feels comfortable
and safe, and it’s really just coasting along, playing the greatest hits.
It’s the Eagles of sessions.
So, I decided that I needed to shake things up a bit.
Kids are talking about this new AI stuff.
I don’t want to be the cranky old bastard in the back
ranting about how no one makes good technology anymore.
I need to keep my mind open and attend these weird AI sessions.
Chris, are you familiar with Cognitive Behavioral Therapy: CBT?
Chris: Yeah, that’s what CBT Nuggets is about, right?
Ned: Is it?
No.
Is that really what CBT stands for?
Chris: No.
Ned: I don’t know, man.
I never watched any of that crap.
Chris: [laugh]
. Ned: [laugh] . It wasn’t crap, but I also didn’t watch it.
Chris: I’m sending them an email.
You’re in trouble.
Ned: I didn’t know they were still going concern, to be honest.
A core premise of CBT is to face your fears
instead of avoiding them, so that’s what I did.
Every single session I attended was AI-related.
I think I went to about ten sessions.
Every single one, AI-related.
No running to the sweet and simple embrace of EC2 instances, or VPC design.
It was going to be mainlining AI for two days.
I have to say, I am glad I did it.
And that’s what I want to share today: what I learned over the two days that
I was there, how I’m feeling about AI now, and what the future looks like.
So, let’s get into it.
Chris: Okay.
Ned: We’ll start with the keynote, which makes sense, right?
Supposed to be the beginning or, you know, the
main thrust and theme of a general conference.
You can watch the keynote if you’d like.
It was presented by Federal CTO at AWS, Dave Levy.
No relation to Eugene Levy, unfortunately.
Two quick things about the keynote.
One, the keynote actually happened on day one at 11 a.m.
after sessions had been going on for two hours.
So, I had already attended a couple of sessions,
but I’m going to start with the keynote anyway.
And number two, there was blessedly only
one keynote, as God and country intended.
Chris: Wow.
Ned: Yeah, thank you, sweet baby Jesus, and your tiny cherub hands, one keynote.
From the very outset, Dave made the point that GenAI is changing the world.
He made a reference to the tale of Rip Van Winkle, which is a story
I barely remember, but if I’m recalling correctly, and he explained
it properly, the core conceit is that Rip falls asleep for 20 years
before the American Revolutionary War starts, and when he wakes
up, the revolution is over and the world has changed drastically.
I think that’s it.
Don’t look it up.
The obvious parallel here is that our world is going
through a similar sea change, and the public sector
can’t afford to sleep on it like Rip Van Winkle did.
In fact, he kept using the word revolution
several times to describe what AI is doing.
And the word revolution does feel, like, a little bit loaded, especially
given the political climate, and the seven or eight protesters who
stood up in the middle of his talk, and yelled stuff about Gaza, Israel,
and how evil Amazon is, and questioned how he could sleep at night.
So, I don’t know, maybe find a word that’s a little less evocative.
Maybe?
I don’t know.
Anyhow, it has nothing to do with AI.
Just word choices.
His main point is that GenAI is going to change
the world, that GenAI requires the cloud to work.
And he quoted a study commissioned by AWS that claims
that cloud is 4.1 times more efficient than on-prem AI.
And that AWS has the tools you need to take your AI experiments to production.
Not sure how much weight I put behind a study commissioned by
AWS that finds out, shockingly, cloud is better than on-prem.
But, I mean, what else do you expect Dave to say?
Chris: I have thoughts, but we can—
Ned: Feel free to share.
Chris: Anytime you see a study like this, what do they mean by on-prem?
Ned: Yeah.
Chris: Who set it up?
How much money and time and expertise did they invest in it?
What models are they using?
Et cetera… et cetera… et cetera.
Ned: What
does it mean to be more efficient?
How do you measure that efficiency?
Like, so many questions.
I didn’t want to get bogged down in it.
I will try to find the study if I can.
We will get into AWS’s AI stack in a little bit,
but I do want to mention a couple other things.
First, Dave said something about how AWS
isn’t here to build the best AI applications.
That’s what they expect you to do.
They’re just providing the raw materials, which inadvertently is a dig at
their own Amazon Q product, which ostensibly is a really good AI application.
Whoops.
Chris: Which also should be what AWS does.
Ned: You would think.
Chris: Just provide the raw materials.
Ned: And that’s good.
I actually liked that part of it.
Like, yeah, you give me the raw materials.
That’s what AWS has specialized in for years is giving
developers, here’s a big old Lego set or an erector
set, or whatever you want to call it; build your thing.
You also made an offhand comment about data sovereignty issues,
which I think was a subtle dig at Microsoft’s latest stumble
with handling GDPR and the Scotland police force, I think?
Dave had two guests with him on stage, one at a time.
The first one was the director of AI for the CIA,
and the second was the head of AI for the army.
And both chats turned into weird recruitment pitches for
the CIA and the army, along with a plea from the army
guy for input on their AI layered defense framework.
What was strangely absent from both conversations was any
mention of how the CIA or the army are using AWS for anything.
Chris: That’s classified.
Ned: I guess?
But also, like, you would think if you have guests up on stage at an AWS
event, they’re going to at least mention how they’re using AWS in some way.
Chris: I mean, counterpoint.
This was hosted in DC, so it could very well have been, somebody
called AWS and explained to them that these are going to be the guests.
Ned: Mmm.
Could be.
I mean, the CIA probably has the leverage to do that.
So, since they didn’t say what AWS is doing
with AI, let’s talk about AWS’s AI portfolio.
It’s kind of an open secret—not even a secret—it’s just known that
Microsoft has a first-mover advantage when it comes to AI products.
They partnered closely with OpenAI.
They rolled out Copilot long before AWS had
CodeWhisperer and GitLab had their product called Duo.
Amazon and Google both looked to be caught flat-footed when
it came to AI, and they are both furiously trying to catch up.
That being said, a first-mover can have a big advantage,
but it also means that you’re likely to fuck up a lot.
You’re the one blazing the trail, you’re not following some
well-trodden path, so the chances of making a logistical
or tactical error are high, even if your strategy is sound.
Sometimes being a fast-follower can actually work in your favor because
you can see what the first-mover stumbled over and adjust accordingly.
And I kind of get the feeling when watching AWS talk about
their various AI offerings, that’s kind of where they’re at.
They’re keeping an eye on Microsoft, and going well, as
long as we can stay relatively close behind, we’ll be fine.
So, they don’t have a monster model like OpenAI’s GPT4o.
I guess it’s called four-oh—why did they abandon the semantic versioning?
I don’t understand.
Anyway—they don’t have that running in their data centers.
Instead, they have their own foundational model which is
called Titan, a model that, by most accounts, is pretty mid.
Meh.
Chris: More like a Teen Titan, am I right?
Ned: Ahh, I like it.
All right, which Team Titan would it be?
Chris: The boring one.
Ned: Robin.
It’d be Robin.
Chris: [laugh]
. Ned: [laugh] . But we repeat ourselves.
What is a poor cloud hyperscaler to do?
Build Amazon Bedrock?
Chris: The Flintstones thing?
Ned: Yeah, yeah, they invited Fred over and
just, like, hash it out over some giant ribs.
Chris: And strangely, he did a cigarette ad.
Everyone was deeply confused.
Ned: [laugh] . Oh, hell of a callback.
Well, done.
So, when Bedrock was announced, most people were like, “Huh?” But essentially
what it is—and actually find it pretty interesting now that I understand it—it’s
essentially an abstraction layer on top of other existing foundational models.
So, it is not an LLM.
It’s not a model in and of itself.
It doesn’t tie you down to a single model.
Instead, you can choose to use one or more of their
various supported models, or bring your own model.
Bedrock currently supports Titan, Anthropic, Meta, Mistral, and a bunch more.
You can look up the current list; I’m sure it’s constantly changing.
And the first session that I attended was actually a builder session
where the presenters walked through using Bedrock in a Jupyter Notebook
to interact with four different models, basically, simultaneously.
I very much jumped into the deep end of things, inadvertently.
I [laugh] walked in, they had, like, code up on the
screen, and I was like, what did I just get myself into?
All right.
The idea was to test and evaluate the responses from each model,
and figure out which one would be best suited for a set of tasks.
And you can go back and check that on a regular
basis because the models are constantly changing.
Inside of the Bedrock SDK for Python, there’s a method called Converse,
and this method and the service behind it abstracts the required
inputs and parameters for each model into a consistent set of inputs.
So, every model has its own specific parameters, and the way that
it expects you to structure requests, and the way that it structures
its responses, especially when you’re interacting programmatically.
The key idea here was you can use this Converse method in a method
call to loop through as many models as you’d like at once, and it
has a consistent interface in front of those models, so you don’t
have to adjust your parameters and your inputs, and you know what
form the response is going to take because it normalizes it for you.
Chris: Hmm.
Ned: That seems kind of useful.
If you need to, you can put in model specific-parameters in a separate field,
so in your configuration of the method, you can say, “For this model, add
these two or three customized parameters.” So, you can still add that level
of customization in there, but if you don’t want to, you don’t have to.
And since the technology and the models are evolving so rapidly, having
a low switching cost between models is going to be a huge benefit, versus
just hitching your wagon to OpenAI and hoping for the best from Sam Altman.
I don’t feel super positive about that [laugh]
. Bedrock also has a couple other neat things.
One is something they’re calling Knowledge Base, which
is essentially RAG, aka, Retrieval Augmented Generation.
Like the name implies, you can load up additional
documents in S3 and feed them to a generic model to help
it better understand your unique business and requirements.
Each model has a different way of doing RAG, so it’s
nice to have Bedrock just handle that abstraction.
Hey, I want to use these six different models.
Send it in the document way they prefer from
this S3 bucket that I’ve loaded things up with.
The other interesting concept—and I’m still trying to wrap my brain
exactly around how it works—but it was this concept of agents and tools.
And I’m not talking about agents in the
sense of chatbots, or digital assistants.
Instead, it’s a way that Bedrock lets you define tools
that models can pass work off to in their application flow.
So, the tools are really just Lambda functions that interact with some
other API or data source, and the agent is the part of Bedrock that
does the tool invocation and passes information back to the model.
So, you can ask your application to do something like parse a bunch of
emails for sales orders and then add them to your ordering system, and
the model can use natural language parsing to understand the mail content,
and then invoke the tool to actually interact with the ordering system.
This is the sort of thing that’s going to put a lot of interns out of work.
Chris: Yeah, it’s basically fancified RPA.
Ned: Yeah, that’s a great way to put it.
It is very much like—yeah, whatever RPA stands for, Robotic Process Automation?
Chris: There you go.
Ned: That’s the one.
I got there.
But yeah, if you have existing APIs and other systems that you want
your language models or your foundational models to interact with,
you don’t have to write that interaction into your application.
You just define this tool spec—I think they called it a tool spec;
it something along those lines—and then implement that tool spec in
Lambda, and now your foundational model can interact with whatever
API you want to hand it off to, if it’s internal or external.
I was like, “Oh, that sounds pretty awesome, but also possibly
incredibly dangerous because now we’ve allowed AI to just do whatever
the hell it wants.” [laugh] . I’m sure that won’t come back to bite us.
So, that’s Bedrock in a nutshell.
There’s more to it, I’m sure, but those are the pieces that jumped out to me.
AWS also went to great pains to say they do not store your requests,
the model responses, or the augmented data anywhere inside of Bedrock.
All of it lives inside your account, and the models that you bring with you.
That’s nice.
I think they went to special pains to explain that because of the audience.
Chris: Right.
Ned: So, from there, I want to jump ahead to a presentation that was about
using GenAI in DevOps, which was a topic that I found very interesting.
The two presenters, Tyler Lynch and Pranusha Manchala—I’m hope I’m
getting that vaguely right; apologies to Pranusha—they lead the
chalk talk and polled the audience about issues and challenges they
encounter at all stages of the software design and deployment pipeline.
If you’ve never attended a chalk talk before, they basically have a
whiteboard, and, like, a loose presentation, but it’s very audience-driven.
So, they’re asking for input: what is your experience?
What are you seeing out there?
As we rolled through the issues, they tried to give
examples of how AWS’s various developer services could
help alleviate some of the challenges people were feeling.
So, I mean, you’re at an AWS conference; the solution is going to be AWS, right?
Chris: One would think.
Ned: Yeah.
They highlighted three products.
One was Amazon Q Developer.
Q Developer used to be called CodeWhisperer.
I don’t know if that was better [laugh] or worse.
Chris: I think you do know.
Ned: I do know.
They’re both terrible.
It’s very similar to what GitHub Copilot does.
It plugs into your IDE, and it provides suggestions and a chat interface.
Do you want to know what a chunk of code does
when you’re trying to get to know a codebase?
Highlight it, ask Q to explain it to you, and it will look at
it and say, “This is what this function does,” in plain English.
Do you need to refactor some code to be more
efficient or scanned for vulnerabilities?
You can do both of those things, making
suggestions for how you could rewrite the code.
Q can also write unit and integration tests,
something that basically no one ever wants to do.
As someone who’s working on developing a Go tool right now, I can tell you
that I spend more time writing unit tests than I do writing actual code.
It sucks.
Chris: Not great.
It’s not great.
Ned: Yeah.
Tyler also mentioned that Q can help with code migrations to
newer language versions, and this was really interesting to me.
Amazon used this functionality to migrate from Java 8 to Java 17.
Now, as someone who is not a Java developer, I wasn’t
really sure, okay, but I guess that is a significant
jump, and those are two largely implemented versions.
I guess there’s not a ton in between.
I don’t really follow the numbering.
But basically, they said they had a thousand applications
that use Java 8 internally at Amazon, and so they took Q and
turned it loose on those thousand applications to upgrade
them all to Java 17, and they were able to do it in two days.
Eyebrow-raising indeed.
Chris: [laugh]
. Ned: That’s bananas.
If you think about the amount of work that would be for developers
to update the code and run the tests for it, they saved a
tremendous amount of time, and also washed away a tremendous
amount of tech debt they had being on this older version of Java.
They also have a similar path to get from Java
4 to 8, and then 17 for much older applications.
I feel like Java 4 was current when I was developing Java back in, like, 1998.
They also are working on similar upgrade functionality
for .NET to .NET Core, and Python 2 to 3.
This is the kind of thing that no one ever wants to do.
No one ever sits down, they’re, like, “Today’s the day when we’re
going to pay off our tech debt and upgrade from Python 2 to 3.”
Chris: Yeah.
For most companies, that day is never.
Ned: Right because you look at it, you’re like, there
is no immediate business value in me doing this.
The application still works, I am not doing anything until I absolutely have to.
But doing that upgrade actually does come with some benefits.
Another session I attended was put on by GitLab, and
they were talking about their code assistant, which is
called Duo, and Duo can do refactoring from C to Rust.
And they talked to a public sector organization whose name they
couldn’t say, but they talked to one, and they had this C-based
application that was a bit of a bottleneck for what they were doing.
They had Duo port it over to Rust.
And this was just an experiment.
It was supposed to be like a three-month-long migration process.
They had Duo try to do the migration of C to Rust,
and move it to running on Lambda instead of EC2.
Duo and the senior developer managed to do it in two days.
Chris: It’s this the rule?
Everything happens in two days?
Ned: Apparently [laugh] . The resulting application was 90%
smaller, ran a hundred times faster, and could scale by 50x.
In part because of Lambda.
Chris: Well, and just because of Rust.
And something interesting about this is it’s not just simple refactoring,
but you’re looking at optimizing how the code does what it does.
You know, like, the difference between a junior programmer writing this
program for the very first time and somebody that’s got 35 years of experience
picking up that code and saying, “This is what it should have looked like.”
Ned: Exactly.
And I think that was the point that the presenter made was, senior
developers will actually get some of the most benefit out of this because
they can look at the code that is produced, and they understand the
context of the original code, so they have a better chance of making
sure the code actually works after that conversion, and works properly.
It’s just that this GitLab Duo is going to save
them a ton of time in doing the actual code writing.
So, that was pretty impressive to me.
He also talked about how so many companies will do, like, a lift-and-shift,
and then never actually refactor the code, and so you have all these
inefficient applications eating up cycles on EC2 in AWS or wherever you
decide to host your code, if you can go back and use one of these code
assistants to refactor things pretty quickly—you know, instead of a six-month
lead time, you’re now talking about a two-week lead time, maybe—that’s
actually worth doing, and could be game-changing for some organizations.
Chris: Yeah.
I mean, a lot of times, people simply hide bad code behind larger instances.
Ned: [laugh] . Sometimes?
Sometimes it’s doing a lot of lifting there, buddy [laugh]
. Chris: [laugh]
. Ned: Yeah, the fact that we can just throw more hardware at it, and have
done that for years, this is actually an opportunity to slim down your
code, maybe make it safer—because if you’re moving from C to Rust, it’s now
memory safe at least—and maybe squash some security bugs in the process.
Like, that’s a win for a lot of teams.
Moving on from the code assistant, the other tools
that were mentioned are DevOps Guru and Code Guru.
Terrible names.
The former does scanning of CloudTrail logs looking for anomalous
events, and the latter does code scanning, testing, and assessment.
From what I understand, Code Guru charges by
the line scanned and is prohibitively expensive.
Chris: Okay.
Ned: DevOps guru, similarly expensive for looking at CloudTrail logs.
But they mentioned them, so I guess I should, too.
The last part of the AWS GenAI sandwich is hardware.
AWS has some hardware to throw at the problem, including their
own in-house developed chips called Trainium, and Inferentia.
Chris: Trainium?
Ned: Trainium, which are for model training, and inference, respectively.
But I think you already guessed that.
Are these chips any good?
No idea.
Are the names any good?
No.
They also have Nvidia A100s and soon will have H100s, so maybe
that’s a better bet if you’re doing this model training thing.
Chris: Just a thought.
Ned: Yeah.
Another session I attended was all about the training process
for a foundational model called Falcon, and to train the model,
they use SageMaker, and ECR, and S3, the biggest version of
the model that they train, they did three versions of it.
The biggest one was called 180B, which I guess is for billion parameters.
This thing used 3.5 trillion training tokens, 46,000 petaflop days—I’m
not even sure what that means, but that’s what they quoted—and 4096 GPUs.
The training process took six months, and used P4d
EC2 instances that each have eight A100s in them.
This was not cheap… by any stretch of the imagination.
They actually, through this process, hit the performance limit
of S3, and had to write new custom libraries that go beyond the
3500 PUTs and 5500 GETs per second that S3 normally supports.
Chris: I was going to say, S3 is not supposed to have limits.
Ned: It does [laugh]
— Chris: [laugh]
. Ned: But you have to work real hard to get there.
They did it.
Congratulations.
It turns out, if you use—there’s a version of S3 that is a single
zone—not single region, but single zone—S3—I forget exactly what it’s
called—if you pick that version of S3, the limits go up because it no
longer has to do any replication or spreading across availability zones.
Chris: Right.
I think that version, when you look at it in
the drop-down, is called don’t use this for DR.
Don’t—for God’s sake—use this for DR.
Ned: Yeah, it’s intended for exactly what they were doing,
which is these large data models that they’re loading.
Chris: People are going to use it for DR, aren’t they?
Ned: They absolutely are.
So, that session was super cool because they had AWS and folks
from Falcon’s company—it’s called TII—talking about the model
design, the training methodology, the data sources and efficiency.
I feel like that could be its own episode, so I’m just going to stop there.
Chris: Cool.
Ned: Geez.
I will brief—
Chris: You said you were going to stop.
Ned: I was going to stop talking about that.
I’m briefly going to touch on AI security, just to mention that I
attended a session that talked about how to implement—or security
things to worry about when it comes to interacting with LLMs, and
they mentioned that OWASP has a top ten for LLMs, which I had no idea.
That report was brought together by 500 experts and 125 contributors,
and came up with a list of possible vulnerabilities for LLMs.
I read through part of the report.
It is fascinating.
Once again, this paper could be its own show, and I would
like to have someone from OWASP or one of the contributors
come on and explain it to me like the third-grader that I am.
Chris: I support this plan.
It means I have to do less work.
Ned: Well, there you go.
Me too [laugh] . So, just to bring it all
down to brass tacks, am I an AI convert now?
Was drinking from the firehose enough to bring me over to the AI side?
No.
My goal at the conference was to enter each session with an open mind.
I did my best not to scoff at ridiculous claims, I tried to take
people and the services at face value, to listen, to not judge.
And for the most part, I was able to turn off the supercritical and
cynical parts of my brain, and just kind of absorb it as it was presented.
Now that I’ve had a little time to digest, here’s what I think:
for starters, GenAI is still very much in its early days.
There were several comparisons to the dotcom bubble, and I think that’s fair.
There’s an embarrassing amount of money being poured into anything AI
right now in the same way that any company that had a vague association
with the World Wide Web was showered with funding in the late-’90s.
That bubble must and will pop.
There will be some genuine use cases that are worth pursuing,
some that are a little ahead of their time, but they’ll come back
around, and some that are just absolute crocks or complete scams.
A lot of them are that.
The most compelling use case I saw—and you may have already
guessed this—was around developer and ops productivity.
There’s a metric ton of tech debt at every large company, little
desire to actually fix it—well, I won’t say ‘a little desire.’ There’s
desire to fix it, but not enough time or money to fix it—this may
actually be the best use case for GenAI AI as it exists right now.
GenAI is good at understanding code.
It’s good at migrating it, it’s good at testing it, and it’s
good at ingesting a lot of data and making it available.
So, it’s going to help Dev and Ops tremendously.
Although I have a little concern around junior talent, and them
being developed if we’re going to end up replacing them with AI.
So, that is a potential problem.
Everything else was AI whitewashing.
Walking the expo floor, basically, every
single booth had AI plastered on it somewhere.
I am not exaggerating.
Every company, from storage to monitoring to professional
services, had AI something, and 90% of them were talking about
some kind of shitty chatbot or some basic-ass machine learning.
In a few years, it will be something else
in the same way that cloud was at one point.
But most of it is just bullshit.
Am I glad I took the plunge?
Yeah.
It was informative to see what people are really
doing versus the marketing and the general messaging.
I can see there’s a lot of potential for disruption coming down the
pike, but it’s not nearly as dire or revolutionary, as some suggest.
Or I don’t know, Chris, maybe I’m wrong and AGI
will be born next week and put me to the sword.
Whatever.
I’m going to be on vacation.
My digital assistant can handle it.
That’s it.
Hey, thanks for listening or something.
I guess you found it worthwhile enough if you made it all
the way to the end, so congratulations to you, friend.
You accomplished something today.
Now, you can sit on the couch, query Developer Q—Q
Developer?—for the best way to rewrite your code, and just relax.
You’ve earned it.
You can find more about this show by visiting our LinkedIn page,
just search ‘Chaos Lever,’ or go to the website, chaoslever.com
where you’ll find show notes, blog posts, and general tomfoolery.
We’ll be off next week because I actually am genuinely going on vacation.
So, you’ll have to wait two weeks to see what fresh hell is upon us.
Ta-ta for now.
Chris: When are you not on vacation?
Ned: I’m sorry, I was on vacation.
What?
[laugh]