Phil: Hello everyone, and welcome
to another episode of APIs.
You won't hate the podcast, it's
the audio version Instead of blog
posts, we're here talking about APIs.
And today I've got Joel Claremont
with me, who is well known around
the Laravel PHP community but
might be new to some of you folks.
So Joel, Hello.
what's going on?
Joel Clermont: Not too much.
Yeah, I've listened to some
of your episodes here and I'm
like, I am definitely not in
the API dev tooling space.
That is not my specialty.
, I'm coming at this more as the,
the, in the trenches practitioner
building, shipping, supporting
APIs and not, not like the higher
level tooling person that's
building things for other people to
Phil: Well that's, that's fine.
So we're not, we, we have, because I
have been working in the kind of open API
dev tooling space, a lot of the content
ends up being quite specific to that.
And that's not really what this
podcast is meant to be like.
If you cast your
mind back to way back when of
the origins of this, of APOs, you
weren't hate as a an anything before
it was a community in a podcast.
I was writing that book.
Build APIs you won't hate, which was
pretty much how to make rest Dish APIs
in Laravel and all of it was Laravel.
So
it's lovely to be chatting to Laravel
people and I still build APIs in
Laravel and I think it's important
that we don't, like, I don't want to
kind of separate out that there's the,
the API community and then like all of
the, the programmers who are actually
building APIs in various frameworks.
They're all, it's all overlap overlapping
circles.
And, whenever I'm talking to people
about like.
One of the main things in, in API derail,
I always try and explain to folks is that
like if you go to API conferences and
you only ever talk about APIs and you
sell your products to other people who
care about APIs, it's a very small world.
But if you go to Laravel, PHB meetups and
and things like that, then there's actual
people who might end up using your stuff
who have never heard about any of the
wonderful, lofty things you've been
talking about and really should have.
So that's kind of why I'm talking
to you today of, yeah, you're
posting about like how, how you
like to make API specs with Laravel
or how you don't, so you wrote an
article called I have It right here.
Why I Don't Like Generating API Specs,
which is written nail on the head.
do you wanna tell me a
little bit more about this?
And you said there's a series of
blog posts and like what led to this?
Tell me more.
Joel Clermont: Yeah, it, it actually
started like, I think the, so, so, first
of all, for a little context, I, I have
a site called mastering laravel.io and
I publish, for some reason, I committed
to a daily newsletter Monday through
Friday, and I, I cross post to the site.
But so any, any random thought that
pops into my head, I'm like, oh,
that could make an email, right?
So I'll, I'll, I'll write it down.
And I was working on a project recently
building an API for a Laravel company.
And we were writing the spec and,
and the client, like they, they're
an existing product and they're
like, we've never written a spec.
Like we only have an internal team.
I.
And so that kind of started
the discussion, which is
like, I think it's useful.
In fact, I know it's useful to have
a spec even if you don't have a giant
team, even if you don't have like a
separate frontend and backend team.
I think like even if it's a full
stack small team or a single dev,
it, it is nice to have a spec.
It, it sort of keeps you honest.
And so that, that was the, the start
of the whole discussion in, in the
newsletter, but then it ultimately led
to, okay, hopefully I've convinced you.
Now how do I get started?
And where most developers like to go
is, well, let's use code to do it right?
Like, that's what we do.
And I personally have never had
success with that approach outside
of like demos and a proof of
concept or maybe like breaking
ground on something to get started.
But the code first approach.
Has been problematic for me.
And so that, that was, that was where
my experience was coming from and I
was trying to share that, like, it's
not that complicated to like just write
the file yourself and, and I think
there's actually some benefits to it.
Phil: I mean, so many developers are
running about, like, everyone has
to learn Kubernetes and everyone has
to learn these 25 different things.
Comically.
Most of those do end up
being slightly overloaded.
yaml DSLs.
But the idea of learning this
one other yaml DSL is completely
unacceptable to anyone
and we'll never do it.
And, That's pretty funny.
So just in case, there are
people who haven't heard me
banging on about open API in the past.
Joel says spec, he means API
specification, which is one other
term for like API description
or API design document.
There's a million different names of
this stuff, but yeah, basically it's
like.
I'm gonna, I'm gonna describe the
endpoints and properties and parameters
and request bodies and all the bits
of the, the interface of the API with
some sort of machine language, which.
tools can use generally for docs.
Pretty handy for mocks
and loads of other stuff.
And I've built a spectral API scanner.
So literally you put your open
API in and it goes, oh, that's
gonna get you hacked buddy.
Watch out for that.
There's a million things you can do with,
with an API spec when you've got it.
But yeah, when, when
left are the developers.
I'm picking up on what
you said a minute ago.
Like the, if you leave it to the
developers they're just gonna
write a whole bunch of code.
And if you, if they ask you, if
you ask them for documentation,
they'll go and write down like 10
lines somewhere and it'll be bad.
So at some point over the last I.
I don't know, eight five.
Over the last five years specifically,
I feel like open API has picked
up in popularity massively.
And that's partly 'cause people like
me bang on about it all the time.
But it's bloody useful for
everyone to know what's going on.
And you've got that machine learnable
format, so you, and, and you've got this,
this spec where you can generate some
docs and you can use it for contract
testing in a million things to, and
that helps keep those docs up to date.
But some folks don't like
to write.
That YAML out by hand.
'cause that is pretty boring.
There's a bunch of other ways of doing it.
There was gooeys and things, but people
don't like to write that by hand and
instead they try a bunch of creative
ways to avoid having to write that.
You were talking about in
Laravel, there's a few kind
of Laravel specific tools.
How, how do, how do
people do it in Laravel?
How do they get, how do they get
their code spitting out open API
so they don't have to write it.
Joel Clermont: Yeah, it's what you might
expect if, if, even if you're not in
Laravel, you know, where, where there's
some, some element of static analysis
and then that can be enhanced with
some sort of annotations or like inline
documentation to, to enhance it further
or to specify like examples of things.
Phil: Mm.
Joel Clermont: and I probably
should say like I've.
At least not super recently.
I've never actually tried to use
one of those packages in earnest.
Like, I will try, somebody
will recommend one, and I'll be
like, oh, I'll give it a shot.
And I usually get a couple days into it.
I'm like no, this is, it is just, it
ends up, you're just conforming to a
different layer of abstraction in the
sa in the same sort of specification.
And so, it's, it's
really a personal thing.
Like I'm sure there are teams out there
that do this and it works great and
I'm sure there are use cases where it,
it's beneficial, but I'm generally not
working on massive, massive projects.
You know, like, like maybe there's 50
to a hundred API endpoints and a lot of
times there might not
even be a public consumer.
Is it
okay.
I mean, it's not stripe.
Phil: right now, but yeah.
Fair enough.
Joel Clermont: Okay, well I'm
counting like get versus post too,
right?
So, you know, in terms of
resources it's probably a lower
number, but, but yeah, it's, it's
not, I don't work on teams of like
50 developers and you know, that
like, so there's probably some
efficiencies that become more.
Important and beneficial where the
trade offs might skew the other
direction toward code first tooling.
IJI just am not there and
I here, maybe this is a
question for you, Phil.
Are there any tools that you've
seen that do spec generation,
that actually leverage the spec?
Like things like refs and, you
know, things like that Or is does
it end up spitting out some really
flat, verbose document that no
human would ever write in that way?
Phil: Yeah, that's the problem.
Most of those tools will just kind of
just vomit something upwards, which is not
particularly.
Helpful.
I mean, it generally, it, that
sort of stuff doesn't really
matter if you are just putting it
straight into a documentation tool.
'cause the documentation
tools will flatten it.
so yeah, the, the way, the
way that it seems to me is.
Somebody has said, Hey, we've,
we've started using open API to
have one kind of documentation
hub, a developer portal, whatever.
We've got this, we've got this thing,
and we need all the teams to make open
API and we don't care how you do it.
I've been that guy.
I've, I've done that.
And I'm like, you know, some of them are
like, oh, we've got it all in Postman,
so we'll convert it into open api.
I, I'm like, sure, whatever.
And some of them are like, oh, we've
got open API two or Raml or these
different old kind of defunct formats.
And I'm like, whatever, I can
convert that for you and, and or, you
know, and then eventually you end up
with open API and everyone's like, great.
Some of it's weird and some of it's ugly
and some of it's not entirely right.
But that's a step one is we now have one
documentation hub for the entire thing.
Great.
But then over
time it's a case of like, well, how
do we get this to be more useful?
And.
And how do we use this for
more verticals as well?
Like if you are just shoving it
into docs, then it kind of makes
that nice stripe, like three coal
API stuff that everyone wants.
But
if you try and use it for, you know,
code generation and a bunch of other
use cases, it, it kind of falls apart
and you can't it gets a bit hard
to like add stuff to it as well.
So.
the one thing I've seen, which helps, if,
so, if you've got a API spec that's being
kind of just a magic appeared out of your
API one thing, that can be pretty handy.
Have you heard of open API Overlays?
Joel Clermont: Mm.
No.
Phil: It's this like extension thing.
So open APIs started doing like working
groups or special interest groups.
And these special interest
groups create different things.
So they're doing like workflows so you
can document entire kind of workflows
of, of API calls and things like that.
And one of them is overlays
where you can basically.
It's kind of like a spectral rule set.
If you're familiar, you kind of
have like a J path and you're like,
go to this bit of the document and
then like add this or remove this.
And so you can theoretically have some
horrendously vomited out, open API and
then be like, and I'm gonna add some
descriptions in here, which are like
10 paragraphs long and have a bunch of
markdown so you can have your technical
writers kind of wedging that stuff in.
And so.
My, my initial like, no one should ever
do code first ever, because then you have
this useless thing is, is somewhat kind of
abated a bit where it's like, okay, you,
you've, you've given us this useless thing
and now some technical writers can jump
through some quite horrendous hoops to
make it a bit less useless, but that still
isn't like the most
shining endorsement for it.
Joel Clermont: you know, like you
mentioned, if if, if the only people
reading the spec are doing so via
like a generated HTML portal, then
it, it doesn't probably matter
internally if it's using like refs for
shared schemas and things like that.
But
I, you know, just like I'm thinking and
recommending you write it by hand at
least once, at least know how to do it.
I also.
I think like, I like to look
at the spec by hand too.
Like
the, the tool, the tooling is great
to auto generate fancy, nice branded
documentation, but I, I do find some
value looking at the spec directly and
then especially seeing, oh, this, you
know, this specific type of request
or this, you know, form payload is
used more than one place.
Like that tells me something about
the overall API design that you don't
see if it's just all flattened out.
Phil: Mm.
Yeah, I mean, that's a useful thing.
So kind of having intent.
Written somewhere else outside of
the code, not only helps you spot
obvious mistakes from a wider pool
of people that don't necessarily
know how Laravel works but
it, it just becomes a, it
becomes a lot more useful.
Like you can have a bunch of
people doing design reviews that
literally don't know Laravel and
don't know its random conventions.
And, and it also means that if
you accidentally have a slip of the
keyboard and, and change some code.
You didn't necessarily change all of
the annotations that came with it then.
I, the graphic I used on a blog post
talking about this was like a, a
drinks cabinet with a labeled that as
milk and it's clearly orange juice.
Like people kind of confuse
the like proximity with
accuracy if like these annotations are
somewhat near the code I'm describing.
So I'm sure someone will definitely
remember to update them both.
And so.
Kind of having it in that second
place means you can then kind of you,
you have those extra checks where
you, you use that open API as an
assertion in your contract test that
says, does my code match what I think
it does and what the design review
team think it does, and what everyone
involved so far has thought it would.
And so that, that for me is kind of
why I'm a big fan of the approach.
But it's, it's just really hard to get
a PA developers to do another thing.
They're like, I'm a coder.
I write
code.
I would like to write some code for
this.
And yeah, it's, it's a weird one.
Joel Clermont: One of the benefits
I think there is in writing the
spec separate from the code is like.
Especially if you start with the
spec is it forces you to think
through the requirements in a level
of detail that you don't often do.
You know, if you're getting a
requirement from some stakeholder
and they're like, Hey, I want an
endpoint to do this, this, and that.
Okay, well, like who can do that?
You know?
How long can that field be?
Like there's all these constraints that.
Somebody has to make a decision
on, and like that spec becomes the
forcing function to decide like,
well, what is the maximum length?
What, what is the format
on this date time?
Or, you know,
just things like that that
otherwise you might get to later
in the middle of implementation.
And a dev just like, well, I
guess we'll just do it this way.
And it wasn't actually
discussed or decided on.
Phil: That's a really good point.
That's I think where I was heading was
that like the API spec being thought of as
a documentation mechanism is kind of the
problem here because it's it's basically.
It's not always a problem.
There are, there are use cases
where you have an API and you
need to like play catch up, right?
There's the there's,
there's API design first.
There's API code first, and the,
and I've done polls where it's
like, are using API code first.
API design first.
Switching from one to the other
or some awkward combination.
And every time I do this, every
couple of years, the answer's
always like 25% of each.
So people, there are some
teams who are doing code
first, some are doing design first,
and, and it's just all over the place.
But yes, like those, those tools that
can help you kind of grab a whole
bunch of metadata out of your code
through static analysis and maybe
you add a few other things in there.
Can really help you get a head start.
And then you have the open
API that you can go back and put
assertions into your code base
to make sure that it's correct.
And then you can expand that.
And if you want to add a new
endpoint, then you can design first
on that, on that open API file.
You can add a new endpoint that's
completely imaginary, start working on it.
Have everyone agree whether
that's a good idea or not.
Whether it solves,
solves the needs or not.
Run it through a mock generator,
a mock server so that people can
interact with it before it even exists.
And then later on.
Code it and make sure that it
matches the contract test for that.
But do you think, do you think
that's how everyone should work?
Joel Clermont: I like your use
case because I, I do join a lot
of, I'll call 'em legacy projects
where you don't have documentation.
In fact,
I hate to say this, sometimes
you don't have tests, right?
Like,
and so that I could see as a useful
time to generate something just,
just to get a starting point.
And then edit it from there, and then
like, you know, write contract tests
around the little bits as you go.
And, and so I, I gotta remember
that because a lot of times when
I join a project, like decisions
have already been made and now
you're just trying to catch up.
And I, I think code generation could
be an excellent use case for that.
Just to, to get you to
where you wanna go faster.
And to not have to write all of that
by hand and, and reverse engineer it.
You know, like try to
figure out, well what are all
the overlapping validation rules
or schema constraints in the
database layer or things like that.
A static analysis can get you 80% there.
Phil: for sure.
I think.
An interesting third way I've come
across, and we've written about
it a bit on the blog, is kind of
using HTP traffic inspection, right?
So I've, I've definitely
inherited APIs where no one
knows how the heck this works.
You've gotta like decompile the iPhone app
and there's some third party client
that you can't even look at the
code of and, and like no one can
get into the compiled Java app on
some bloody API that people haven't
got the password for anymore.
It sounds silly, but I've been in these
silly situations that help out a lot of
struggling
Joel Clermont: boy.
Phil: And everyone's like, right,
we've gotta rebuild this API.
And you're like, what API if I can't
find any of the endpoints, like,
everyone, check your browser history.
What are we doing here?
And so being able to
kind of wrap it in a, in a
traffic proxy that essentially
says like all the requests.
Going in, or you can literally, you
can wedge them into the browser.
And, and then kind of run
around clicking things.
There's, there's a bunch of these
different tools for sniffing actual
traffic going through the system.
And then
you can kind of take all of
that and output an open API
file and say, there you go.
And I feel like if we're, if we're
talking kind of the catching up
with design first, then I'm almost,
I'm mu I'm, I generally feel more
interested in going with the traffic
sniffing approach because it's real.
I.
Like you can still look at some, look
at some stuff and, and get it wrong.
Like if, if when you've got these
complicated dynamic payloads where
there's this or this or this, that
could come through and you're like,
it's definitely only this 'cause you've
only seen it come, you know, you haven't
noticed any null values in there.
So you said it, it is a required property.
Mistakes like that can be
made depending on how much
the sample size goes through.
But I, I feel like the promise
of having these kind of framework
integrated static analysis.
Tools is potentially misleading
unless they can hook into enough
where you literally don't need
to add other stuff yourself.
'cause as soon as you ask the humans
to get involved, that's when they're
just introducing human error.
Have you come across anything like that?
Joel Clermont: Yeah.
Um, Not directly.
'cause again, I've not worked on a project
recently that has done the
automated tooling, but, but I've,
I've experienced drift for sure.
And, and honestly that's one of my fears
with Code Generation, I, you alluded
to it earlier, is like if you make
an unintentional change, well, and it
just regenerates the spec to match it.
Now your tests are still passing, but I.
Unless a human catches that in code
review or design review of some sort,
like you could unintentionally change
your API even in a breaking way,
which to me is like the worst possible
outcome of of, of, of an API change.
So I don't know if there's solutions
to that, but yeah, the, either
the intentional or unintentional
drift is just, just an ongoing
problem I have experienced,
With that traffic inspection like
that's super useful for like sort of
those black box scenarios you were
describing.
Like, I'm gonna have
nightmares about that.
But I, I'm almost wondering too if
like that in conjunction, like when
you're, when you're, when I'm joining
a project, like I'm coming into some
project, if I, if I just use static
analysis, that doesn't tell me.
How much dead code is in there,
right?
And like now, maybe I'm wasting a week of
work trying to document and test something
and then I, ship it to the client.
They're like, oh, we haven't
used that in five years.
Like that.
Well, that was just a
wasted week of effort.
So I could see pairing the static
analysis with the traffic analysis
just to like, even to prioritize.
These are the paths that
get the most traffic.
Like
let's focus on those.
These endpoints, they never got traffic.
Oh, I'll ask the client even
before I start working on it.
Oh yeah.
We can just delete that.
Right?
Like, so I could see a hybrid
approach there being useful too.
Phil: Yeah.
Interesting.
The I've seen a few frameworks that are.
me love the idea of code
first a little bit more.
I wrote
an article recently and I can't remember
the names of any of these things.
I think it's like Hu Hugo Oh,
Hummer in, in Node and Fast.
API I.
I'll get some problem links in
the show notes when I remember.
But there, there are, there's
this kind of new idea of like
open API Aware frameworks, right?
Like built.
Into the core of the framework is
in the past, in the past you've got
like swagger, PHP, which I hated.
Like I, I, when I, when I first got
involved with swagger, when I first
tried to start trying to figure out
how to document an API, I looked
at swagger, PHP, and went, ah, for
all the reasons we've discussed is
literally you put a bunch of doc block
annotations in your code
and hope that it's correct.
And, and I ran away and then
I ended up doing other things.
But that's still like the main premise.
But with modern tools, like like Scribe
is a really cool tool and like props
to the people that have made this.
But they, I.
They've essentially got rid of quite a few
of the annotations because the framework
is kind of saying that this is a poster
or, or a get, and like they've, they know
what the URL is from looking at the code.
They're doing a lot of that
static, static analysis,
but it's still kind of hacked
into the framework a bit.
And there's still a certain
amount of duplication.
And so what I like about
the frameworks that are.
Coming at this from like the
framework itself will tell you what
your open API is, because just by
using the framework, we are building
up all of the bits that you need.
That seems really cool.
Like when, when you write
the validation rules for your
models, they all line up with an
appropriate validation in open API.
So when you're doing that, it sounds
that that, that to me seems like it
would be pretty much just fine because.
The, the main, the main thing of like a PA
design first is it's more about planning.
You should like plan this thing and
spend a lot more time in the planning
stage past the whiteboard, but before
you start writing loads of code.
But if the framework can
make itself simple enough
that it, the framework is, is.
Just like a config DSL where you are,
instead of writing open API in yaml,
you're writing open API in this kind of
a bit of PHB code and it's easy enough to
read and it makes sense, and then you just
shove a bit of business logic in there
later that, that to me would be fine.
That kind of absolves the differences,
removes the differences between open API.
Well between design first and code first.
It kind of makes them
become the same thing.
You just kind of like add your
business logic some other time.
But none of the tools in Laravel
that I've seen are that, and it's
quite rare to actually find them.
And at this point I worry that those
projects are kind of just like hobby
projects and not a whole thing.
So
there's a bit of hope in there, but
I don't think anyone's taken the
idea too seriously at this point.
Joel Clermont: Yeah.
The, the reason this, this came
on my radar recently is there is a
newer package, like it's not even
at 1.0 in Laravel called Scramble.
I
don't, I don't know if
you've seen this one.
It will look at, like in, in LAR you have
a form request with validation rules.
It will statically
analyze those rules and.
Set like max links or field
types or things like that,
purely based on that with like
zero annotation, which is, I,
I do like the promise of it.
I really do.
Like I, I get the appeal.
Um, one, one other thing too
I was thinking about is I.
You know, when you mention whiteboards
and upfront design, like I cringed a
little bit because like I, I've come
over the years to recognize like you
never can plan super far in advance.
And, But I, you know, so one thing
I wanted to clarify is like, even
with the design first approach,
I generally do it like literally
end point by end point, right?
So I, I design it, I write the spec,
we agree on it, I implement it.
Then we repeat for the next one,
not like, well, let's design the
entire API and, and see what happens,
because that, that just
never materializes correctly.
Phil: man.
Yeah, it's, it's something that I really,
I need to do a whole episode on this,
on like, how do you actually plan an
API, because the stuff I said in my book
was, was pretty immature at the time.
I think I was like, just think
of all the endpoints you need and
then like post and get next to
'em or whatever is, is a bit dumb.
But, but like the phases, the phases,
of of planning and that's like
literally stakeholder interviews,
which is like a, a, a concept from
user research and, and, and ux.
and then kind of taking everyone's
needs and requirements, and not
necessarily too literally, but
distilling down the essence of what.
They want and what they need and, and
how multiple different stakeholders
can ask for different things.
But finding the middle ground that
will be just fine for everyone, like
doing all this really complicated work.
And then another group of people or,
or you, with a different hat on, can
then start to say, well then these
are the end points that I need.
But even then, like, is the,
is jumping straight into
endpoint endpoints premature.
Like, do, do you need to think a little
bit more about, about kind of the
workflows and the, and the type of API
that could be used and like, should
this even be a, a endpoint based API
or should, should it be event driven?
There's, there's so many stages to the
planning and it's hard to figure out.
Where you should start and open
API and design the spec first.
Stuff like it doesn't help
you with any of any of that.
Like you, literally, your planning
starts with Slack calls and whiteboards
and like Miro boards and sequence
sequence flow diagrams and like
chatting to a load of people and
having a spreadsheet of feedback and
all this crazy stuff that no one,
no one that I know is really working
on, apart from the tools that I've
mentioned, but they're like generic.
And then once someone has says.
Someone has said, you definitely
need to make a HT PAPI and it
should probably do these things.
That's when it's time
to get the open API out.
'cause you're literally turning the words
they've just said right there of like,
make an API that does this, this and this.
Then you kind of are going, well okay,
we'll have that end point, that end
point and that then end point and.
Where design first gets really handy
there is you can say, okay, I'm gonna turn
those words into, like, I'm gonna reshape
them into a little bit of yaml and I'm
gonna like, the A PO will look like this.
Then it's much easier to change that
little bit of YAML than it is to go and
change a whole pile of Laravel code.
You're like, oh God, now I've got rename
all of my controllers and that means
I've gotta rename all my models and all
the relationships are gonna be broken.
And I, you know,
that's, that's really daft.
And so
pretending that.
Code first is quicker 'cause you just add
some annotations to some code that you
wrote is predicated on the fact that you
wrote exactly the right code first time.
And when does that happen?
Joel Clermont: It doesn't.
Right, exactly.
At least not for anything.
Non-trivial.
Phil: Yeah, well, alright.
Pretty decent agreement on that.
Screw.
Screw that.
Joel Clermont: Awesome.
Phil: That was
Joel Clermont: See, I I we're far
too reasonable, Phil, in our, in
our older age and maturity we're,
we're mellowing and we're, we're just
seeing things much more reasonably.
Phil: Absolutely.
Absolutely.
I haven't had a fight about framework
interoperability in a long time.
Well, fantastic.
Let's, let's call that where
can people find more of you?
You are, you're all over the
internet doing all sorts of things.
What, what what are you up to?
Joel Clermont: Yeah, my, my main hub of
activity now is mastering laravel.io.
You know, on there we have
a podcast for Laravel and PP
devs, the, the mailing list,
some books and courses if you
wanna, like, keep me fed, like,
you know, all, all that good stuff.
But that's, that's really the
central point for all of it.
Phil: Nice.
I will have to I need to get on
that newsletter because I am using
Laravel more and more and they've
gone, they've gone on like a rampage
of major versions and I'm like, I, I
remember thinking four and five are
quite nice and now version eleven's out.
I'm like, I don't know what's happening.
Brilliant.
Joel Clermont: Yeah, it
does tend to move quickly.
Phil: Cool.
Alright, well thank you very much and
yeah, pleasure having you on the show.
Joel Clermont: Yep.
Talk to you later.