Subscribe
Share
Share
Embed
The Product Engineering Podcast explores how software teams can move beyond shipping features and start delivering measurable outcomes. Hosted by engineering leader James Charlesworth, each episode covers practical ideas for product engineering, engineering leadership, team operating models and building products that create real impact.
Using AI to build a prototype of
something isn't the same as building
production code that will
last and stand the test of time.
Often you can get something 80% of the
way there using AI, and that extra 20%
could take you many, many times as long
than that first 80% took.
So don't fall into that trap of thinking
that just because you can build a
prototype that looks amazing, you've got
something you can ship to production.
(Music)
Hi, I'm James Charlesworth.
Welcome back to the
Product Engineering Podcast.
And today we're going to
talk about prototyping.
And this is really one for the engineers
because I know a lot of engineers have
been expressing this opinion for a while
now, ever since AI came around.
AI is really, really good at creating a
nice looking product, but the step
between getting that code like out there
and looking nice on the user interface,
and then getting it into production and
being able to support it into production
is not as simple as it first seems.
Now, software engineers absolutely know
this, but product managers and people
that aren't necessarily super, super
technical from a coding point of view
think that just because it
looks right, it is right.
Now, that doesn't necessarily mean that
having an 80% of the way
their prototype is a bad thing.
In fact, as I'll argue many times on this
podcast, being able to create a prototype
that looks good and basically functions
is an extremely valuable learning tool,
but it's not the same thing
as writing production code.
So let's break down what some of those
gaps are and some of those things that AI
still can't necessarily do.
And we'll work out where you can use
prototypes and where you can use AI and
where you should
probably do things properly.
And in fact, it's probably worth going
back in time to before AI existed because
this problem has
always basically been there.
Ever since things like React and stuff
like that came out, it's been fairly
trivial to create a nice looking user
interface, maybe with some mock data, and
then put that in front of people and say,
this is what the
product is going to look like.
And then a lot of people fall into the
trap of thinking that that really quick
prototype you made is
ready for production.
If anyone has ever been to a hackathon,
hackathons usually revolve around the
smoking mirrors effect of I've made
something that kind of
looks like it's going to work.
This is directionally where we want to
go, but we haven't
actually written production code.
It's extremely difficult even in the age
of AI to write proper hardened production
supportable code in like a small amount
of time, like a one day hackathon.
And nothing's really changed there.
AI has just turned like a one day
hackathon into a 10
minute prompt exercise.
Ultimately, it hasn't changed the fact
that the code that you've written that
might look brilliant isn't always ready
to go into production.
And so what are those gaps?
If you've built something using claw code
or you built like a user interface from
scratch, what are the gaps between what
you've built and what looks amazing and
what's actually production ready?
Well, there's many of them and they're
often things like scalability, cost.
A lot of the time in AI will just not
really care about how much something is
going to cost to support in production.
So it might give you a super expensive
solution that isn't necessarily going to
be commercially viable.
There's also things like observability
and maintainability.
So good software engineers will know that
in order to support some software into
production, you need to instrument that
software with observability and metrics
and things like that.
So that when something breaks in
production, you can very, very quickly
work out what's going to break.
Potentially, you can work out as it's
going to break before
something happens in the product.
All of that observability is hidden to
people that are using software.
If you're a user of a web application,
you probably don't realize quite how much
instrumentation and observability is
built into that on the back end.
And so if you're being shown a prototype
of something that was written with AI,
you don't know whether or not there's a
way for the person who built that to
actually monitor what's
happening on the back end.
And to predict when things are going to
fail and when it's going to crash.
But you do need all that.
In order to support something in
production, it needs to be observable.
It needs to be secure.
That's another thing
that often gets missed.
It needs to be scalable.
You can build things for
different levels of scale.
People, it's a mistake to think that
scalable is a yes or no decision.
Code can be scalable to the point where
you want it to scale to, or it can be
scalable to like a larger point.
But you very rarely get a situation where
code will scale from zero to infinity for
the same trade-offs.
Unless you've been really, really
conscious about giving the AI information
about how much you want it to scale,
which a lot of people won't do, then what
you'll end up with is a piece of software
that's designed for a certain level of
scale and you try and implement it to a
different level of scale.
Scale works both ways.
So if the AI has given you a piece of
software that works for a million users,
but you've actually only got 100 users,
that's not necessarily a good solution.
That's going to cost
you a lot of money to run.
The infrastructure is going to be
extremely complicated and expensive, and
you've only got 100 users, so you're not
getting that benefit.
Equally, if it's built you something that
will only work for 100 users and you try
and roll it out to a million users, it
will break there as well.
So you need to be able to
design for the correct scale.
And again, that's one of those hidden
pieces of architecture that isn't
apparent in something that
was just vibe-coded with AI.
So you can have a product that looks
functionally brilliant, but all of those
questions around scalability and security
and observability and maintainability,
all of those are hidden and you don't
know what those decisions were or if
those decisions were even
made in the first place.
So it's really important for everyone
that's building software rapidly with AI
to just be aware of those kinds of
decisions that need to be made in order
to make something production-ready.
So you have the user interface that
you've built and you've maybe got some
back-end functionality, then ask yourself
how are we going to make sure this
scales, how are we going to make sure
it's secure, how are we going to support
it in production, what are we going to do
when something goes wrong, where the data
have actually been stored, which the
parties are using, all the legal stuff.
All of those kinds of questions need to
be answered in order to take your
prototype into production.
Now that doesn't necessarily mean you
shouldn't do it, you absolutely should
build prototypes because sometimes you
don't need all of those other questions
to be answered in order to
get value out of a prototype.
So a prototype that's been built with AI,
if you're working on a product
engineering team and you're working in
quite an unknown environment, an AI-built
prototype can give you something that's
more valuable than anything else,
probably more valuable than the
production code and it can give you
answers, it can give you learning.
So if building a prototype allows you to
explore an idea or it allows you to
answer a really pressing question about
the correct path forward in your product,
then you should do it.
If you want to put that into production,
then you need to actually answer all of
those other questions.
So you need to make it scalable, you need
to make it secure, you need to have
observability and all
of the other things.
Even at a code level, it can be useful to
make sure that you've got the right
coding practices and the code's clean and
readable and doesn't duplicate itself and
all the kinds of things that software
engineers care about, that's what goes
into production product and that kind of
stuff is completely invisible to the
person using the prototype.
So the solution to this is just to make
it explicit what is included and what is
not included in some
software you've written.
If you've built an application very, very
quickly to test out a concept, then just
make it explicit that this might not be
100% scalable, it might not be
maintainable, that's okay, as long as
you're saying that that's why we've done
it, then you can still get the learnings
and the value out of that product.
Two pieces of product might look exactly
the same, two pieces of software could on
the surface appear exactly the same, but
one of them is production ready and one
of them is not and you wouldn't
necessarily know from the
outside whether they are or not.
So that's where the skill
of engineering comes in.
So if a product manager, for example, is
prototyping something with AI, then the
role of an engineer on their team is to
help them guide them down the right
questions they need to answer before
that's ready for production.
Now I spoke on a previous episode about
context and accountability.
If somebody is making a decision to build
something inside your product, then you
do really want to empower them to be able
to make that decision.
But they don't always know the context
behind whether that's
the right decision or not.
So it's the job of the organization to
create a structure in which people that
know about security and observability and
scalability and support, those people can
provide that context to the people that
are going to make the decisions about
whether or not it should be built.
So ultimately, if you've got a product
engineering team with software engineers
that know all about the architecture and
the correct way to scale something, and
they've got product managers that maybe
know the correct thing to be building,
bringing those two together and allowing
them to share that context will allow you
to build things that are
initially not production ready.
And then once you know it's the right
thing to build, you go
make them production ready.
So it's ultimately a two stage process,
you build something that looks finished,
and then you go and make
it finished afterwards.
And those two things can be very, very
different levels of effort.
Sometimes the amount of effort that goes
into building an AI prototype is more
than you think it is.
So just because something looks like it
was vibe coded with a one shot prompt
doesn't necessarily mean that that person
hasn't spent a lot of time
exploring different ways.
So don't disregard the amount of effort
that goes into building an AI prototype
because they can sometimes involve quite
a lot of learning, quite a lot of
iteration, quite a
lot of experimentation.
There's a lot of back and forth talking
to an AI is still working.
So don't disregard the fact that
somebody's got a good prototype, just
because it's not production ready.
But at the same time, don't disregard all
of the work that you need to do to make
some software production ready.
It's really, really important not to have
this smoke and mirrors effect, where
you've built
something that looks amazing.
And then everybody just
assumes it can go into production.
Sometimes things can, but they won't
always go into production in the way that
you necessarily need them to.
There are a lot of platforms out there
that will give you a in air quotes
production ready piece of software.
But that doesn't necessarily mean that
it's the right production ready piece of
software for your organization, for the
level of scale that you're going for, for
your particular regulatory
or security requirements.
None of that sort of stuff is guaranteed
just because something looks production
ready, or just because you're using a
tool that has told you
it's production ready.
So ultimately, it comes down to a balance
between over
engineering and under engineering.
Something that looks amazing, but doesn't
have all of the scalability and the
production ready stuff
in it is under engineered.
But something that is super, super
scalable and really observable and
supportable in production, but doesn't
necessarily need to be at this point,
that will be over engineered.
If all you're trying to do is learn
something, if you're trying to learn if
this is the right feature to be building,
then you don't necessarily
want to over engineer something.
It doesn't need to be fully scalable if
you're only going to try it
out with a handful of users.
So don't over engineer stuff.
Don't allow software engineers and people
that really want everything to be super
production ready to slow you down.
Don't allow those people to stop you
trying out prototypes and things with a
subset of users with
the outcome of learning.
But equally, don't allow people that have
built a prototype to just ship it to
production without thinking about all
those other things as well.
The most important thing you can do
practically is to make the definition
explicit between production
ready code and a prototype.
So when you've got something that's been
built, really genuinely document what is
stopping this thing going into production
today, and what do we
still need to build?
You don't want every new prototype to
have a list of checked off items saying
it's ready for production because
probably that prototype took way too long
to build and it's not going
to give you effective learning.
But obviously you don't want a prototype
that has a bunch of unchecked boxes
because it's not ready for production.
So if this is something that resonates
for you, if you've seen people trying to
ship prototypes to production when
they're not ready, or you've seen people
over engineering a prototype because
they're being told it has to be
production ready when it doesn't
necessarily have to
be, then do let me know.
My email address is
james at train2code.com.
I really welcome feedback on this product
and this podcast, and I really, really
want to have other
opinions and other voices on this.
I am booking guests for the future, so if
you want to guest on this,
then reach out to me as well.
My name is James Charlesworth, and this
is the Product Engineering Podcast.